On 2026-11-05 Microsoft Foundry blogged that Foundry Local CLI 0.13.0+ can run 4-bit Qwen3.8-27B (~18 GB weights) on-device via ONNX Runtime and connect it to GitHub Copilot through Bring Your Own Key. Needs an NVIDIA GPU with at least 24 GB VRAM (about 64K context on 24 GB; up to the model's 262K max on 32 GB+). Vendor-cited model-card scores include Terminal-Bench 2.1 73.0 and SWE-bench Pro 61.7; Microsoft also reports higher generation speed than same-quant llama.cpp on RTX 5090 and DGX Spark. Mac and AMD support are still on the roadmap.
Key Takeaways
- ✓Shape: Foundry Local 0.13.0+ runs 4-bit Qwen3.8-27B (~18 GB weights) and connects to the GitHub Copilot app via BYOK
- ✓Hardware: NVIDIA GPU with >=24 GB VRAM; ~64K context on 24 GB, up to 262K on 32 GB+; install CUDA/cuDNN yourself
- ✓Vendor-cited model-card scores (not a 4-bit re-run): Terminal-Bench 2.1 73.0, SWE-bench Pro 61.7, NL2Repo-Bench 42.3, DeepSWE 1.1 42.2
- ✓Stack: ONNX Runtime + DFlash2 speculative decoding + INT4/KV quant + prefix caching; Microsoft says the gap vs llama.cpp widens past 131K tokens
- ✓Limits: Microsoft says cloud models still win on hard reasoning; Mac/AMD not shipped; Copilot CLI local model does not imply offline—set COPILOT_OFFLINE separately
Key Decision Metrics at a Glance
Turn your technical choice into a development budget
Compare 40 dev plans & simulate token costs vs $20/mo subscriptions
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Core Background & Industry Pain Points
Cloud coding agents bill per token, ship repo context off-machine, and stop offline. Local ~27B coding hits VRAM and long-repo throughput walls. Microsoft's pitch is hybrid: everyday edits on-device, cloud models for hard reasoning.
Architecture Highlights & Internals
Foundry Local 0.13.0+ serves 4-bit Qwen3.8-27B (~18 GB weights) via ONNX Runtime with DFlash2 speculative decoding, INT4/KV quantization, and prefix caching. About 64K context on 24 GB VRAM; up to 262K on 32 GB+. In the Copilot app, add Foundry Local as a BYOK provider using the local endpoint from foundry server status.
Authoritative Benchmarks & Measured Scores
The blog cites model-card scores—Terminal-Bench 2.1 73.0, SWE-bench Pro 61.7, NL2Repo-Bench 42.3, DeepSWE 1.1 42.2—not an independent 4-bit re-run. Microsoft also reports faster generation than same-quant llama.cpp on RTX 5090 and DGX Spark past ~131K tokens (vendor-measured), and notes local models stay weaker on complex reasoning.
Developer Hands-on Guide
winget install Microsoft.FoundryLocal, foundry model load qwen3.8-27b, then wire the local endpoint in Copilot Model providers. Fit: privacy-sensitive, offline, cost-stable everyday coding; keep cloud for heavy refactors. Check network traffic and COPILOT_OFFLINE, and smoke-test tool calling on the quantized build.
Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.