LMSYS published a technical deep dive on a megakernel written from scratch in CUDA C++ and PTX (about 8,000 lines) for Meta's open-weight Muse Spark Open. Each decoder layer runs as a single kernel launch that overlaps compute, memory loads and cross-GPU communication. On 8x GB300, single-request decoding goes from 517 tok/s on the SGLang baseline to 1,109 tok/s, about 2.1x, and each decode step drops from 12.9 ms to 5.9 ms. The code is open source as an SGLang plugin at facebookresearch/sgl-muse-mega (MIT). It only covers batch size 1 and is experimental, so it buys latency rather than cheaper tokens.
Key Takeaways
- ✓Twice as fast: on 8x GB300 at TP8 with one GSM8K request, decoding goes from 517 to 1,109 tok/s, about 2.1x, with the same draft model and acceptance length
- ✓Far fewer launches: kernel launches per decode step drop from about 1,450 to 97, one per layer
- ✓Step time: 12.9 ms to 5.9 ms; verify goes from 10.9 to 5.3 ms and the draft from 1.16 ms to 0.44 ms (175 launches down to 8)
- ✓Speculative decoding: the DFlash V2 draft guesses 15 tokens per step; the model verifies 16 and accepts 6.6 on average, versus 8 verified and 6.3 accepted for Xiaomi MiMo-V2.5-Pro
- ✓Scope: batch 1 only, with other requests falling back to stock SGLang; eight top GPUs serve one user, so this sells latency, not lower cost
Key Decision Metrics at a Glance
Turn your technical choice into a development budget
Compare 40 dev plans & simulate token costs vs $20/mo subscriptions
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Background and the problem
A normal serving engine runs a model as a chain of small kernels. LMSYS counts about 1,450 kernel launches per decode step when SGLang serves Muse Spark Open, and the GPU drains and refills between them, leaving SMs and warps idle. Even with speculative decoding, the single-request baseline reaches only 517 tok/s. For latency-sensitive uses such as live coding assistants and voice, that idle time is the lag users feel.
Architecture and how it works
Author Pranjal Shankhdhar wrote about 8,000 lines of CUDA C++ and PTX. Each launch covers everything from one layer's attention to the next layer's qkv projection and KV write. Every SM runs one block of 12 warps with specialized roles for loading, tensor cores and communication. Weights are prefetched into shared memory or L2, and the 14 experts most used on the previous step are pulled into L2 before routing finishes. Cross-GPU handoffs poll the data itself instead of using costly memory fences. The post also found that oversized kernels thrash the roughly 128 KB instruction cache, so each launch first warms its own code into L2.
Benchmarks and measured results
All figures are LMSYS's own measurements: 8x GB300, TP8, one GSM8K request, median of 20 steps. Decoding goes from 517 to 1,109 tok/s and step time from 12.9 to 5.9 ms, with verify dropping from 10.9 to 5.3 ms and the draft from 1.16 to 0.44 ms. The DFlash V2 draft lets the model verify 16 tokens per step and accept 6.6 on average. Built-in timestamping costs about 2.8% of a step and can be compiled out.
Getting started for developers
The code is at GitHub facebookresearch/sgl-muse-mega (MIT) and runs as an SGLang plugin enabled with --enable-nvidia-megakernel, currently targeting GB300 (sm103). It only covers batch-1 decoding, other requests fall back to stock SGLang, and the project is marked experimental. Next steps are larger batches and a move toward a GPU "operating system" that schedules work at run time.
Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.