CMU and Fudan Pioneer Looped Model Foundations with Huginn MoE: Sandwich Topology Matches 32B Feedforward Baseline with 8B Resident Weights

Background and the Problem

As parameter counts swell, local edge devices and on-device coding agents are bottlenecked by physical VRAM limits. Looped language models (such as NVIDIA Dory 1.0 or Ouroboros architectures) repeatedly cycle tokens through a compact set of shared physical Transformer blocks, mimicking the capacity of ultra-deep networks within a fraction of the parameter memory footprint. However, prior architectures empirical designs hopelessly entangled recurrence placement, input injection mechanics, and latent state representations. Researchers from CMU LTI and Fudan University presented Part I of Towards Looped Models Done Right (arXiv:2610.23036), establishing the first principled framework for looped transformers.

Architecture and How It Works

1. Prelude-Recurrent-Coda Sandwich Topology (Huginn):

• Proves that making the entire model an unbroken loop is mathematically sub-optimal. Instead, the team proposes a P8-R12^8-C8 sandwich: 8 untied prelude layers extract basic syntactic representations, a compact 12-layer recurrent block cycles 8 times (yielding 96 logical layers), followed by 8 untied coda layers for task decoding;

1. Input Injection and Recurrent-State Mechanics:

• Demonstrates that continually injecting prelude-derived representations across recurrent passes prevents representations from collapsing and eliminates vanishing gradients;

1. Scaling to Looped Mixture-of-Experts (Looped MoE):

• Extends the sandwich topology to sparse MoEs (25 physical experts, Top-2 routing per step), establishing Huginn MoE.

Benchmarks and Measured Results

Trained on a rigorous 500-billion-token (500B) corpus (Tables 2, 5, 6):

• 4x Memory Footprint Compression (Table 6):
• The feedforward reference D112 MoE contains 32.0B resident parameters / 3.2B active parameters;
• Huginn MoE has only 28 physical layers, requiring just 8.0B resident parameters / 793.9M active parameters (a 75% reduction in disk and VRAM weights);
• Matching Frontier 32B Performance (Table 5):
• Across MMLU, BBH-CoT, GSM8K, and MATH500, the 8.0B-resident Huginn MoE matches the 32B feedforward MoE while decisively outperforming naive unstratified loops (Ouro MoE);
• In dense 730M-parameter experiments (Table 2), the sandwich architecture delivers optimal Pareto efficiency across ARC-C, HellaSwag, and CommonsenseQA.

Getting Started for Developers

• Architectural Reference: Detailed derivations and configuration tables are provided in arXiv:2610.23036.
• Edge Deployment Strategy: Teams targeting local on-device agents on consumer GPUs (16GB–24GB VRAM) can leverage the sandwich looped MoE pattern to achieve 32B-class reasoning from an 8B physical memory footprint.