Cambridge Reveals LLMs Encode Knowledge in Hidden States Before the Unembedding Layer Can Express It, While Univ of Copenhagen Isolates Personalized MoE Experts via Crosscoders

Core Background & Industry Pain Points

In LLM pretraining dynamics and behavioral personalization, two foundational questions remain unresolved: First, when does a language model actually acquire a capability during pretraining? Standard developmental analyses track when the model's output probabilities first emit the correct token, conflating what the internal hidden states encode with what the unembedding (readout) matrix has learned to express. Second, when personalizing LLMs for individual users, dense Transformers suffer from severe neuron polysemanticity—where each neuron entangles multiple unrelated concepts—making it difficult to localize, interpret, and causally steer the components governing user preferences.

Architectural Highlights & Core Mechanisms

Two new mechanistic interpretability studies from the University of Cambridge (arXiv:2610.21025) and the University of Copenhagen (arXiv:2610.22127) leverage sparse crosscoding to answer these questions:

• Cambridge's Parameter-Trajectory Crosscoding (arXiv:2610.21025): Unlike activation-based sparse autoencoders, parameter-trajectory crosscoding fits a shared sparse dictionary directly to every token's unembedding readout row across training checkpoints using only checkpoint weights. Comparing readout dictionary evolution against hidden-state linear probes separates internal capability acquisition from readout expression.
• Copenhagen's Paired-Activation Crosscoders for Personalized MoE LLMs (arXiv:2610.22127): Adapts three open-weight Mixture-of-Experts (MoE) LLMs on a large-scale personalization dataset and trains crosscoders on paired activations of the personalized and base models. This isolates interpretable personalized latents and maps them directly to the specific routed experts encoding distinct user preference features.

Authoritative Benchmarks & Empirical Comparisons

• Locating the 0.4%–1.1% Pythia Readout Window & Closing the Expression Lag (arXiv:2610.21025): Across Pythia pretraining checkpoints, parameter-trajectory crosscoding pinpoints a narrow early window—just 0.4% to 1.1% into pretraining—where the unembedding matrix reorganizes and WordNet lexical categories become decodable from its rows. By that point, linear probes on hidden states already decode task information that the model's own outputs fail to express. Pretraining controlled models from scratch that differ solely in the readout layer's learning rate proves that raising the readout learning rate causes the unembedding layer to develop earlier and closes the hidden-to-output gap sooner with negligible impact on final loss.
• Lower Polysemanticity & Causal Expert Steering in MoE LLMs (arXiv:2610.22127): Across three open-weight MoE families, personalized MoEs match dense LLM quality at a fraction of active parameters while exhibiting significantly lower polysemanticity than dense models with matched active parameter counts. Causal interventions confirm that the discovered crosscoder latents suffice to recover personalized behavior, and boosting the routing weights of their associated experts directly steers the non-personalized base model toward the target user persona.

Developer Practical Guide & Quickstart

Pretraining researchers can use Cambridge's open-source weight-trajectory toolkit at https://github.com/hematteo/learning-to-read-out to audit checkpoint unembedding dynamics and tune decoupled readout learning rates in AdamW/Muon schedules. Applied AI teams building personalized assistants can adopt open MoE backbones and apply expert-routing biases to dynamically toggle user personas at inference time.