Hugging Face Ships Ultimate MoE Training Handbook: ~500k–800k GPU-Hours Distilled into a Megatron-LM Guide

Core Background & Industry Pain Points

Top open models are mostly Mixture-of-Experts, yet reproducible large-scale MoE recipes remain scarce. On 2026-11-05 Hugging Face and author Nouamane Tazi released the free Ultimate MoE Training Handbook; co-founder Thomas Wolf amplified it on X.

Architecture Highlights & Internals

Built around Megatron-LM: Chapter 2 covers hardware checks, image build, tokenization, dense baseline and a first single-node MoE; Chapter 3 covers routers (softmax/sigmoid/sqrt(softplus), fp32), expert sizing and shared experts, hiding all-to-all cost, aux loss vs DeepSeek-style loss-free bias, dense upcycling, plus a recommended flag block. About 55 interactive figures and runnable code ship with it.

Authoritative Benchmarks & Measured Scores

This is an engineering handbook, not a new model board. Thomas Wolf cited ~800k GPU-hours, ~70k words and 55 figures; author Nouamane separately cited ~500k GPU-hours—prefer the Space body; no independent third-party scores here.

Developer Hands-on Guide

Read at huggingface.co/spaces/huggingface/moe-blogpost. Follow Chapter 2 for a dense control and first MoE, then add Chapter 3 flags against logs. Numbers come from clusters such as Alps GH200 and do not transfer blindly.