NVIDIA Open-Sources SoL-Pi: Coding Agent Inference Acceleration Slashing Token Traffic by 49%

Core Background & Industry Pain Points Long-running coding agents frequently execute 20-50 iterative turns of editing, testing, and debugging on large codebases. In standard stateless serving setups, tens of thousands of tokens of repository context are repeatedly re-sent and re-computed on every turn, causing astronomical API token consumption and sluggish turnaround times despite minimal file changes. ### Architecture Highlights & Internals SoL-Pi introduces a hardware-accelerated agent serving pipeline: a Branching Tree KV-Cache that treats conversational turns and file system states as immutable sub-branches, re-using prior attention keys without re-computation; a speculative draft agent running smaller models for patch proposals; and zero-copy GPU memory streaming that maximizes Tensor Core utilization. ### Authoritative Benchmarks & Measured Scores On the SWE-bench Verified benchmark suite, SoL-Pi slashed aggregate token traffic by 48.7% and accelerated end-to-end bug resolution by 2.08x. Total compute cost dropped by 33.4% without sacrificing solution accuracy. ### Developer Hands-on Guide Developers can deploy SoL-Pi as a drop-in transparent proxy between client agents and OpenAI-compatible inference backends, with pre-configured Docker containers available on GitHub.