Research map / Learning theory and algorithms / Matrix and tensor methods
Models of Heavy-Tailed Mechanistic Universality
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the high-temperature Marchenko-Pastur (HTMP) random matrix ensemble, derived from an entropic-regularization modeling framework, to explain heavy-tailed mechanistic universality in trained neural networks' feature and weight matrices.
Problem
Heavy-tailed spectral densities in Jacobians, Hessians, NTK/activation matrices and weight matrices are empirically robust correlates of model quality, but lack a rigorous statistical explanation of their origin.
Approach
Model trained feature matrices via an entropic-regularized (stochastic minimization) framework yielding an inverse-Wishart-type density; introduce a high-temperature beta-ensemble whose eigenvalue-repulsion parameter kappa controls tail exponents, so power laws arise from data correlations, reduced temperature, and reduced eigenvector entropy/structure without heavy-tailed elements.
Key results
- HTMP ensemble reproduces inverse-Gamma behavior at the left edge and power-law tails at infinity for trained feature matrices, with heavier tails as kappa decreases.
- Proposition derives neural scaling law L ~ n^{-l(2+kappa/(2gamma)-kappa/2)} for ridge regression on activation matrices with HTMP spectrum.
- MiniAlexNet trained weight spectra across batch sizes 1000 to 5 transition through MP bulk, bleed-out, spikes, and HTMP with kappa=5.5 and 1.9, matching the 5+1 phases.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| decrease the MiniAlexNet training batch size | MiniAlexNet · CIFAR-10 weight-matrix spectra | spectral tail heaviness | Helped |
| decrease the MiniAlexNet training batch size to 5 | MiniAlexNet · CIFAR-10 classification | task-learning ability | Hurt |
| decrease the eigenvalue repulsion parameter κ | HTMP spectral model · trained feature-matrix spectra | spectral tail heaviness | Helped |
Surprising findings
- Heavy-tailed spectra can arise without heavy-tailed matrix elements, since the HTMP mechanism operates via eigenvalue repulsion and structure rather than element distributions.
- Trained weight matrices in the MiniAlexNet experiments with very small batch size show severe rank deficiency and are over-regularized, making that case uninformative.
Limitations
- Connection between kappa, alpha, beta and generalization is not made precise; the authors call linking to generalization premature.
- Deterministic equivalence for general covariance inverse-Wishart matrices is unknown, so scaling law derived only for isotropic case.
- Empirical validation is limited to small-scale experiments (1000 CIFAR-10 points and MiniAlexNet).
Experimental setup
Datasets: CIFAR-10
Benchmarks: CIFAR-10
Metrics: empirical spectral density fit Kolmogorov-Smirnov statistic generalization error scaling exponent
Baselines: Marchenko-Pastur law Kesten mechanism explanations population covariance power-law explanations scaling law theories
Builds on: High-Temperature Marchenko-Pastur distribution Heavy-Tailed Self-Regularization theory Neural Tangent Kernel PAC-Bayes theorem Donsker-Varadhan variational formula
Closest papers by meaning
- Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectra (ICLR 2025 · similarity 0.88)
- The Underlying Universal Statistical Structure of Natural Datasets (ICML 2025 · similarity 0.87)
- Emergence of heavy tails in homogenized stochastic gradient descent (NeurIPS 2024 · similarity 0.86)
- Investigating the Overlooked Hessian Structure: From CNNs to LLMs (ICML 2025 · similarity 0.85)
- Scaling Laws in Linear Regression: Compute, Parameters, and Data (NeurIPS 2024 · similarity 0.85)
- Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.