During the SGLang Summit on November 12, the SGLang team and AWS Annapurna Labs released SGLang-Neuron v0.1.0 as open source in public preview. It is an out-of-tree SGLang plugin: set SGLANG_PLATFORM=neuron and the same server and APIs serve models and generate RL rollouts on Trainium, running on native PyTorch through TorchNeuron. AWS also published a Deep Learning Container image. LLMs currently run on Trn3, and diffusion models on trn2.48xlarge.
Key Takeaways
- ✓Open source and license: GitHub sgl-project/sglang-neuron under Apache 2.0; v0.1.0 is a public preview, and features, APIs and supported models may change
- ✓No changes to SGLang: the plugin registers the neuron platform through entry points and is pinned to upstream commit 43881a7; the image tag is v0.1.0-neuron-py314-sdk-2.33.0
- ✓LLM list: 8 architectures including GPT-OSS-120B, the Qwen3 family, Qwen3.5-397B-A17B, GLM-5.2 (1M context) and Kimi-K3 (1 prefill node plus 2 decode nodes)
- ✓Key mechanisms: a fully asynchronous runtime, giga-kernels that run a whole decode step in one NKI kernel call, PD disaggregation over NIXL and EFA, and HiCache prefix caching in host memory
- ✓Diffusion models: FLUX.1-dev, FLUX.3-Action, MiniMax-H3, Wan2.2 A14B and LingBot-World-V2 on Trainium2
Key Decision Metrics at a Glance
Turn your technical choice into a development budget
Compare 40 dev plans & simulate token costs vs $20/mo subscriptions
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Background and the problem
Open-source inference frameworks have long centered on NVIDIA GPUs. Teams that wanted AWS's own Trainium chips usually had to switch toolchains and rewrite serving code. SGLang ran a closed beta on Trainium in October; v0.1.0 is now open source, with the goal that anyone who knows SGLang can use Trainium.
Architecture and how it works
SGLang-Neuron is an out-of-tree plugin. SGLang core keeps the API server, request lifecycle, RadixAttention and prefix-cache bookkeeping, while the plugin implements Trainium-specific models, attention, MoE and the graph runner. TorchNeuron registers Trainium as a native PyTorch device; operations compile with torch.compile, with NKI kernels where performance matters. Every prefill and decode shape is compiled at startup, so the server never compiles while serving. PD disaggregation moves the KV cache over NIXL and EFA, and evicted prefixes stay in host memory and load back on a later hit.
Benchmarks and measured results
Neither the announcement nor the AWS blog gives a unified comparison against GPUs. The README says each model recipe includes measured results and that accuracy is checked with GSM8K. What can be confirmed is capability: Kimi-K3 runs disaggregated across three Trainium3 nodes, and GLM-5.2 supports 1M context. Performance claims should wait for third-party testing.
Getting started for developers
The fastest path is to pull the official image public.ecr.aws/neuron/pytorch-inference-sglang-neuron on a Trn3 instance, launch gpt-oss-120b at TP8 and EP8 as in the AWS blog, and call the OpenAI-compatible /v1/chat/completions endpoint. Building from source needs Neuron SDK 2.33.0 and SGLang at commit 43881a7, or the plugin refuses to launch. This is a preview, so load-test your own workload before production use.
Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.