Xiaohongshu, PKU, and CUHK Shenzhen Introduce LangDiT: Re-Engineering Transformers for Continuous Diffusion Language Models Slashes GenPPL to 11.33

Background and the Problem

Continuous diffusion language models (CDLMs, such as LangFlow and Plaid) represent a promising alternative for non-autoregressive parallel text generation, offering high-throughput generation free of token-by-token latency constraints. However, virtually all existing CDLMs directly transplant architectures from causal LLMs or vision Diffusion Transformers (DiTs). These generic models suffer from feature scale collapse and gradient routing conflicts when diffusing discrete text tokens embedded in continuous manifolds. Researchers from Xiaohongshu (Dots Studio), Peking University, CUHK Shenzhen, and UCLA designed LangDiT (arXiv:2610.22330), the first dedicated transformer architecture optimized specifically for continuous diffusion language modeling.

Architecture and How It Works

1. Decoupling Feature Normalization from Noise Modulation:

• Introduces backbone RMSNorm and headwise Q/K RMSNorm alongside a wider noise conditioning projection (doubled from 128 to 256), decoupling intrinsic representation scaling from stochastic noise updates;

1. Vocabulary Shortcut Detachment & Bounded Calibration:

• Separates the forward shortcut contribution from direct codebook gradient routing at the output vocabulary head, preventing self-conditioning feedback from corrupting base embeddings;

1. Confidence-Gated Training Self-Conditioning:

• Gated bootstrap inputs filter training steps by entropy confidence, ensuring that only reliable latent estimates are injected during self-conditioning while unweighted posterior-mean estimates are used at inference.

Benchmarks and Measured Results

Evaluated on OpenWebText unconditional generation of 1024-token sequences:

• LangFlow Formulation:
• Achieves a state-of-the-art generation perplexity (21.72 GenPPL) at 1024 denoising NFEs, and 44.72 GenPPL with just 128 NFEs;
• Plaid Formulation (Table 2):
• Slashes generation perplexity to 11.33 GenPPL at 1024 NFEs, and 27.51 GenPPL at 128 NFEs;
• These gains isolate the Transformer architecture itself as the fundamental driver of performance, achieved without altering the underlying training objectives or latent representations.

Getting Started for Developers

• Research Reference: Full architectural specifications are detailed in arXiv:2610.22330.
• Engineering Takeaways: Teams exploring non-autoregressive generation, parallel speech synthesis, or diffusion-based code infilling should integrate LangDiT's Q/K normalization and vocabulary shortcut detachment to eliminate gradient instability and mode collapse.