CityU HK Proposes Neural Spectral Capacity (NSC): Zero-Shot Architecture Search & Pruning in 2s on CPU

Core Background & Industry Pain Points In LLM architecture design and structured pruning, allocating representational capacity under strict compute and memory budgets is the fundamental bottleneck. Standard scalars like #Params and #FLOPs fail to capture topological structure: two models with identical parameter counts perform drastically differently based on depth-width ratios, head allocations, and FFN expansion rates. Training-free proxies require loading heavy checkpoint weights and batches of calibration data, relying on expensive heuristic search algorithms that lack global optimality guarantees. ### Architecture Highlights & Internals City University of Hong Kong researchers introduced Neural Spectral Capacity (NSC) grounded in Random Matrix Theory: 1. Marchenko-Pastur Closed-Form Formulation: Under standard random initialization, weight singular values converge to the Marchenko-Pastur distribution. NSC computes an exact spectral scalar directly from network configuration dictionaries in microseconds—without instantiating weights or computing gradients; 2. Layer-wise Additivity & NSC-DP Solver: By exploiting layer-wise additivity, the NSC-DP dynamic programming solver guarantees exact global capacity maximization under budget constraints in seconds on a single CPU; 3. Data-Free Structured Pruning: NSC-DP serves as an optimal layer allocation solver for pruning dense LLMs to target parameters without requiring calibration samples. ### Authoritative Benchmarks & Measured Scores - Fine-Grained Ranking Accuracy: On FlexiBERT architecture pairs differing by <10% parameters, #Params ranking collapses to Kendall tau = 0.082, while NSC achieves tau = 0.505; - Search Latency: NSC-DP discovers a Transformer-XL variant outperforming human baselines on WikiText-103 in 2 seconds on CPU; - LLaMA-7B Pruning: Prunes LLaMA-7B down to 5.7B across eight commonsense reasoning benchmarks with zero calibration data, accelerating search by 5,900x over state-of-the-art training-free proxies. ### Developer Hands-on Guide Install the lightweight nsc package from source. Pass model specification dictionaries directly into compute_nsc or solve_nsc_dp to evaluate architectures and derive optimal pruning configurations. Paper: arXiv:2609.23087, Repository: https://github.com/Optima-CityU/neural-spectral-capacity.