Researchers from MWS AI have published 'GeoPair: Geometry-Preserving Cross-Layer Factorization for Training-Free Transformer Compression' (arXiv: 2609.25963). While transformers exhibit deep inter-layer redundancies, existing post-training compression techniques optimize layers in complete isolation or rely on crude heuristic groupings that obliterate layer-specific activation geometries. GeoPair introduces an optimization-driven, training-free framework that sequentially pairs structurally compatible projections across layers to learn shared dictionary representations. Combined with structured sparsity, GeoPair establishes state-of-the-art compression efficiency across text and vision architectures without functional degradation.

Key Takeaways

  • ✓Replaces isolated layer-wise pruning with mathematically grounded cross-layer geometric compatibility optimization.
  • ✓Completely training-free, executing matrix decompositions in minutes without requiring backpropagation datasets.
  • ✓Learns shared dictionary representations that preserve layer-specific calibration geometries under activation flows.
  • ✓Couples with structured sparsity to compress model VRAM by 35% to 48% with near-zero downstream performance degradation.
  • ✓Theoretical proofs, sequential optimization algorithms, and benchmark scripts published on arXiv and Hugging Face.
🔗

Project Links & Resources

Direct Access
Direct access to official project resources and documentation
🌐
ArXiv 论文预印本 (2609.25963)
arxiv.org/abs/2609.25963
↗
📝
Hugging Face Daily Papers 讨论主页
huggingface.co/papers/2609.25963
↗
🔬

In-Depth Technical Analysis

Core Background & Industry Pain Points Deep transformers exhibit profound cross-layer redundancies, yet post-training compression methods routinely optimize layers in isolation or enforce naive adjacent-layer sharing. Disregarding the delicate geometry of intermediate activation distributions destroys layer-specific calibration geometries, triggering catastrophic performance collapse in zero-shot reasoning and code synthesis under aggressive compression ratios. ### Architecture Highlights & Internals GeoPair replaces heuristic layer merging with an optimization-driven, training-free framework: (1) It scans the entire network depth to identify structurally compatible layer pairings based on geometric projection invariants rather than adjacency; (2) It computes shared-dictionary factorizations that decompose paired weight matrices into a compact shared subspace accompanied by sparse layer-specific projection offsets; and (3) A sequential convex optimization loop converges in minutes on consumer hardware without backpropagation or training data. ### Authoritative Benchmarks & Measured Scores Evaluated on LLaMA, Qwen, and vision transformers: at a 40% weight compression target, GeoPair achieves superior perplexity (improving by 1.84 points over heuristic SVD baselines). On MMLU, GSM8K, and HumanEval benchmarks, compressed models preserve 98.6% of original dense reasoning accuracy while reducing peak VRAM consumption by 41.3% and elevating streaming generation throughput by 1.62x. ### Developer Hands-on Guide Systems engineers can inspect mathematical formulations, dictionary learning routines, and PyTorch reference implementations on arXiv and Hugging Face Papers.