Research map / Architectures and efficiency / Pruning and efficient tuning
Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FARMS, which measures weight-matrix heavy-tailedness by averaging ESDs of fixed-aspect-ratio submatrices to remove aspect-ratio bias, improving TempBalance, AlphaPruning, and TB_Sigmoid across CV, SciML, and LLM pruning.
Problem
Heavy-tailed self-regularization (HT-SR) eigenspectrum metrics such as PL_Alpha_Hill are biased by the aspect ratio of weight matrices, leading to inaccurate layer quality diagnosis and poor layer-wise hyperparameter assignment.
Approach
Partition each weight matrix into overlapping submatrices with a fixed aspect ratio shared across all layers, compute the ESD of each, and measure heavy-tailedness on the averaged ESD (merged eigenvalue series), so that the metric no longer depends on the original matrix's aspect ratio.
Key results
- On LLaMA-13B with magnitude pruning at 0.7 sparsity, FARMS reduces WikiText perplexity from 2029.20 (AlphaPruning) to 413.76.
- On LLaMA-7B with SparseGPT at 0.8 sparsity, perplexity drops from 96.02 to 79.42 (about 17.3% reduction).
- On CIFAR100 with TempBalance, FARMS improves mean test accuracy across five scaling ratios, e.g., ResNet 18 79.35 vs 79.03 and VGG 19 74.03 vs 73.89.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace AlphaPruning's eigenspectrum analysis with FARMS | AlphaPruning + Magnitude pruning on LLaMA-7B · WikiText validation, sparsity 0.7 | perplexity: 231.76 -> 173.49 | Helped |
| Replace AlphaPruning's eigenspectrum analysis with FARMS | AlphaPruning + SparseGPT on LLaMA-7B · WikiText validation, sparsity 0.8 | perplexity: 96.02{ $\pm$1.59} -> 79.42{ $\pm$3.86} | Helped |
| Replace AlphaPruning's eigenspectrum analysis with FARMS | AlphaPruning + SparseGPT on LLaMA-13B · WikiText validation, sparsity 0.8 | perplexity: 47.57{ $\pm$2.64} -> 41.44{ $\pm$1.58} | Helped |
| Replace TempBalance's heavytailness measurement with FARMS | TempBalance on ResNet18 · CIFAR100, mean across five learning-rate scaling ratios | test accuracy: 79.03{ $\pm$0.169} -> 79.35{ $\pm$0.126} | Helped |
| Replace TempBalance's heavytailness measurement with FARMS | TempBalance on VGG19 · CIFAR100, mean across five learning-rate scaling ratios | test accuracy: 73.89{ $\pm$0.199} -> 74.03{ $\pm$0.163} | Helped |
| Use FARMS instead of TB_Sigmoid for layer-wise fine-tuning | DPOT-Tiny · 2DCFD, 100% data | L2RE: 1.078e-2 -> 1.017e-2 | Helped |
| Replace AlphaPruning's eigenspectrum analysis with FARMS | AlphaPruning + SparseGPT on LLaMA-7B · Zero-shot tasks, sparsity 0.8 | mean accuracy: 36.63 -> 37.50 | Helped |
| Set subsampled-matrix aspect ratio to 1.0 (ablation) | FARMS + TempBalance on ResNet18 · CIFAR100 | test accuracy: 79.13{ $\pm$0.158} (Q=0.5) -> 79.53{ $\pm$0.177} (Q=1.0) | Helped |
| Use a 2000 x 2000 subsampling window with 10 sampling steps (ablation) | FARMS + SparseGPT on LLaMA-7B · WikiText, sparsity 0.8 | perplexity: 96.02 -> 79.42{ $\pm$3.86} | Helped |
| Average ESD metrics across subsets instead of calculating one overall ESD (ablation) | FARMS CNN processing on ResNet18 · CIFAR100 | test accuracy: 79.36{ $\pm$0.101} -> 79.53{ $\pm$0.177} | Helped |
| Remove the layer-selection heuristic (ablation) | TempBalance on VGG19 · CIFAR100 | test accuracy: 74.19% -> 73.22% | Hurt |
| Remove the layer-selection heuristic (ablation) | TempBalance on VGG19 · CIFAR100 | standard deviation: 0.159 -> 0.277 | Hurt |
Surprising findings
- Previous HT-SR metrics report higher PL_Alpha_Hill values for randomly initialized layers and for wider models, although all initialized layers should be equally under-trained; FARMS removes this trend.
- Without the layer-selection heuristic, TempBalance accuracy on VGG19 drops (74.19% to 73.22%), while TempBalance with FARMS stays stable or improves.
Limitations
- Submatrix window size, sampling steps, and aspect ratio Q are tunable hyperparameters with sensitivity to choice (e.g., window size 2000 best in ablation).
- Gains on some LLM settings are marginal or reversed (e.g., OPT-1.3B Wanda 0.6, LLaMA-V3.1-8B Wanda at 0.7), and CNN processing requires heuristic per-kernel averaging.
Experimental setup
Datasets: CIFAR100 WikiText BoolQ RTE HellaSwag WinoGrande ARC-Easy ARC-Challenge OpenbookQA PDEBench 2D Compressible Navier-Stokes
Benchmarks: CIFAR100 WikiText BoolQ RTE HellaSwag WinoGrande ARC-Easy ARC-Challenge OpenbookQA PDEBench 2D CFD
Metrics: test accuracy WikiText perplexity zero-shot accuracy L2RE PL_Alpha_Hill mean and standard deviation
Baselines: TempBalance AlphaPruning TB_Sigmoid Magnitude pruning Wanda SparseGPT
Builds on: HT-SR theory TempBalance AlphaPruning WeightWatcher Hill estimator
Closest papers by meaning
- A Deeper Look at the Hessian Eigenspectrum of Deep Neural Networks and its Applications to Regularization (AAAI 2021 · similarity 0.86)
- Unifying Low Dimensional Spectra in Deep Learning (ICML 2026 · similarity 0.86)
- Models of Heavy-Tailed Mechanistic Universality (ICML 2025 · similarity 0.84)
- Eigen Analysis of Conjugate Kernel and Neural Tangent Kernel (ICML 2025 · similarity 0.84)
- A Probabilistic Approach to Learning the Degree of Equivariance in Steerable CNNs (ICML 2024 · similarity 0.83)
- A Tale of Two Symmetries: Exploring the Loss Landscape of Equivariant Models (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.