Research map / Architectures and efficiency / Pruning and efficient tuning

Small Singular Values Matter: A Random Matrix Analysis of Transformer Models

NeurIPS 2025 · Architectures and efficiency › Pruning and efficient tuning · approach: Low-rank compression and merging · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Uses random matrix theory as a zero-information baseline to show that small singular values outside the Marchenko-Pastur bulk of non-square transformer weight matrices align with activation covariance eigenvectors and matter for perplexity and fine-tuning, backed by a linear rank-1 teacher-student model.

Problem

Which parts of the singular value spectrum of pretrained transformer weight matrices store information, and whether small singular values (usually discarded in SVD pruning) carry relevant information.

Approach

Compare weight-matrix singular value spectra with Marchenko-Pastur predictions to locate learned outliers at both ends of the spectrum, check overlap of singular vectors with activation covariance eigenvectors, and test importance by zeroing deciles; explain small outliers with a linear random-matrix student-teacher model whose noise has non-trivial covariance.

Key results

• Removing the smallest decile of singular values from the non-square Down-Projection matrix of Llama3-8B drops GSM8K accuracy from 43.2% to 2.0%.
• For non-square matrices the smallest decile causes a larger perplexity increase than some larger deciles on WikiText for Pythia and Llama.
• On BERT fine-tuned first, removing the smallest decile significantly degrades accuracy on BoolQ, RTE, and SST2, whereas pruning before fine-tuning is largely recovered.

Measured effects reported in the paper

Surprising findings

• Smallest singular values of non-square matrices (e.g., Llama Down-Projection) can be more important for perplexity than some larger deciles.
• Attention-Output and Down-Projection matrices of Llama show little overlap with activation covariance eigenvectors despite the importance of small singular values.

Limitations

• Attention-Output and Llama Down-Projection show no activation-covariance overlap, explanation is speculative.
• Noise covariance for LLM weight matrices cannot be estimated from a single training run, limiting validation of the theory.
• Analysis covers only three models and few benchmarks; the scaling with model size is not clearly established.

Experimental setup

Datasets: WikiText BookCorpus BoolQ RTE SST2 GSM8K HumanEval RULER

Benchmarks: WikiText perplexity BookCorpus perplexity GSM8K HumanEval RULER BoolQ RTE SST2

Metrics: perplexity increase accuracy overlap O_k 3-sigma interval

Baselines: standard SVD low-rank pruning (removing largest or small singular values by L2 n

Builds on: Marchenko-Pastur law BBP transition Martin-Hu random matrix analysis of deep learning

Closest papers by meaning

• Beyond Components: Singular Vector-Based Interpretability of Transformer Circuits (NeurIPS 2025 · similarity 0.87)
• Demystifying Singular Defects in Large Language Models (ICML 2025 · similarity 0.86)
• Understanding Transformers via N-Gram Statistics (NeurIPS 2024 · similarity 0.86)
• Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Data (NeurIPS 2024 · similarity 0.86)
• Born a Transformer -- Always a Transformer? On the Effect of Pretraining on Architectural Abilities (NeurIPS 2025 · similarity 0.85)
• Transformer Layers as Painters (AAAI 2025 · similarity 0.85)