On Oct 20, inference company Eagle released EAGLE-4, the fourth generation of the EAGLE speculative-decoding family. One drafter resizes its speculative tree for the batch size, hardware and workload while keeping the target model's output distribution. Eagle reports 961.6 tokens/s for one request on Qwen3.8-27B with 4x B300. The code ships as eagle-harness v0.1.0 with plugins for SGLang, vLLM and an experimental TokenSpeed backend. Many reposts call it open source, but the official license is source-available, not OSI open source.
Key Takeaways
- ✓Vendor figure: 961.6 tokens/s on Qwen3.8-27B with TokenSpeed, BF16, 4x B300, TP4, one request, T=0 (paper Figure 1(a))
- ✓Repo measurements: vLLM on 4x B300 goes from 241.6 to 722.8 tok/s at concurrency 1 (2.99x) and 1.70x at concurrency 8; SGLang gets 1.91x at concurrency 8 and 1.35x at 64
- ✓AMD numbers too: SGLang on one MI355X gets 2.07x at concurrency 8 and 1.56x at 32 (ROCm 7.2)
- ✓Design: sliding-window draft layer, low-rank candidate head and append-only draft cache; the harness calibrates verify latency offline and grows the tree while gain per cost stays high
- ✓License: Eagle 4 Source-Available; free for noncommercial use, commercial use only under $20M trailing 12-month revenue and outside competing offerings
Key Decision Metrics at a Glance
Not enough VRAM? Compare cloud API and self-hosting costs
Compare 40 dev plans & simulate token costs vs $20/mo subscriptions
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Background and pain point
Autoregressive decoding leaves GPUs memory-bound at small batch sizes. Speculative decoding lets a small drafter guess several tokens that the target model verifies in one pass. The catch is that the best tree shape depends on batch size and hardware, and earlier EAGLE-3, DFlash and DSpark setups used fixed depth and top-k. EAGLE-4 aims for one drafter that adapts to every batch size and GPU.
Architecture
Per the official README and the fal deep dive, EAGLE-4 keeps EAGLE-3's use of target hidden states plus token embeddings, adds a sliding-window draft layer, a low-rank candidate head that only computes full logits for shortlisted tokens, and an append-only draft cache. Before serving, the harness calibrates verify latency on the target GPU and parallelism. At each level it compares cumulative-probability gain against measured marginal cost, widening or deepening the tree only while that ratio clears a threshold, then verifies with a tree attention mask and rejection sampling, so the output distribution is unchanged. Multiple draft layers and MoE drafters are not in this release.
Benchmarks
The headline 961.6 tokens/s (Qwen3.8-27B, TokenSpeed, BF16, 4x B300, TP4, one request, T=0) is a vendor figure. The repo support matrix reports vLLM on 4x B300 rising from 241.6 to 722.8 tok/s at concurrency 1 (2.99x) and 1.70x at concurrency 8, SGLang at 1.91x (concurrency 8) and 1.35x (64), and 2.07x on one AMD MI355X. No TokenSpeed throughput is claimed. Community comparisons against DFlash 2 and DSpark were not verified in official docs and are not cited.
Getting started
Install eagle-harness 0.1.0 from PyPI alongside one of Eagle's patched SGLang, vLLM or TokenSpeed wheels or images; stock engines lack the hooks. The only measured pair today is Qwen3.8-27B with EAGLE-Corp/Qwen3.8-27B-EAGLE4. fal also released a 400-prompt Prompt Enhancement SpecBench for creative workloads. Read the license first: it is source-available, and companies above $20M trailing revenue or building competing inference stacks need a separate commercial license.
Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.