Fixing LLM & VLM Evaluation Blind Spots: Exact Two-Factor MCQ Option-Bias Identity, 61M-Edit semCER for OCR, and 14.5x SAE Feature Reproducibility Scaling

Core Background & Industry Pain Points

Across LLM benchmarking, OCR evaluation, and mechanistic interpretability, three measurement blind spots distort engineering decisions: (1) in multiple-choice evaluations (MMLU, ARC), reordering options changes model answers, yet existing metrics conflate letter preference (Selection Bias) with permutation instability (Order Sensitivity); (2) in OCR and VLM transcription, standard Character Error Rate (CER) penalizes a harmless diacritic slip identically to a semantic entity corruption; and (3) Sparse Autoencoders (SAEs) are routinely trained on hundreds of millions of activation tokens without testing whether functional reconstruction convergence guarantees reproducible monosemantic features.

Architectural Highlights & Core Mechanisms

Three new studies provide rigorous mathematical identities and open-source evaluation toolkits:

• Exact Two-Factor Decomposition of Multiple-Choice Option Bias (arXiv:2610.20993): Proves that under any position-balanced option ordering, the pooled selection-rate standard deviation (pooled SD) equals the exact product (within $5.55 \times 10^{-17}$) of two label-free factors: the Flip Rate (fraction of questions whose chosen option changes across orderings) and Positional Skew (standard deviation of option-letter selection rates among flipping questions), alongside a label-free correction $\tilde{s}(f)$ for random ordering subsets.
• semCER: Severity-Weighted Character Error Rate (arXiv:2610.20461): Preserves CER's character alignment and denominator while weighting each grapheme edit by severity distilled from an LLM judge across 16 languages with no word segmentation required.
• Decoupling SAE Functional Convergence from Feature Reproducibility (arXiv:2610.20680): Trains 4 SAE architectures across 3 LLMs on token budgets spanning 5M to 500M tokens, separating subsampled statistical consistency (reconstruction/probing) from the stricter identifiability conditions required for 1-to-1 feature recovery.

Authoritative Benchmarks & Empirical Comparisons

• 41 Checkpoints × 12 Benchmarks Expose Wrong-Option Locking (arXiv:2610.20993): Across 41 checkpoints and 12 benchmarks, Flip Rate tracks accuracy and uncertainty (Spearman -0.935), while Positional Skew tracks option-symbol priors. Prior calibration mainly lowers Positional Skew, whereas aggregating an ordering with its reversal lowers Flip Rate—partly by settling unstable questions onto the same wrong option.
• 61 Million Judged Edits & VLM Real-Word Hallucinations (arXiv:2610.20461): On pairs where CER and semCER disagree, English and Chinese readers side with semCER significantly more often. Furthermore, when a general VLM makes a character error, it forms a real valid word far above chance (making errors harder to spot), whereas classical OCR pipelines stay near chance.
• 14.5x Jump in SAE Feature Reproducibility (arXiv:2610.20680): While reconstruction loss saturates at small token budgets, strict 1-to-1 feature reproducibility on Gemma-2-2B (BatchTopK) rises 14.5x from 2% at 5M tokens to 29% at 50M tokens, with random sampling favoring reconstruction and diversity sampling boosting AutoInterp.

Developer Practical Guide & Quickstart

Engineers evaluating VLM OCR pipelines can download the 61M judged edits and distilled semCER scorer at https://github.com/nguyenx2t/semcer. Benchmark authors should report both Flip Rate and Positional Skew alongside MCQ accuracy, and interpretability researchers can cut SAE training compute by matching token budgets and sampling strategies (random vs. diversity) to whether their goal is functional reconstruction or reproducible feature discovery.