Research map / Probabilistic and scientific modeling / Molecules and proteins
The Underlying Universal Statistical Structure of Natural Datasets
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes eigenvalue power laws and random-matrix (GOE) spectral statistics of feature covariance matrices for real image datasets and proposes correlated Gaussian datasets (Toeplitz covariance) as a proxy model.
Problem
Understanding the universal spectral structure of real-world dataset covariance (Gram) matrices, including their power-law eigenvalue bulk, RMT statistics, and convergence with sample size, to enable analytically tractable data models.
Approach
Model real image covariance bulk spectra as Wishart matrices with a Toeplitz shift-invariant population covariance (correlated Gaussian datasets, CGDs) whose exponent alpha controls the power law, and show real and CGD data both follow GOE chaotic RMT statistics once sample size exceeds roughly the feature dimension.
Key results
- Real-world image datasets show bulk eigenvalue power-law exponents alpha <= 1/2, while uncorrelated Gaussian data gives alpha = -1; CGDs reproduce the bulk spectra (e.g., KL=0.001 for FMNIST).
- Bulk eigenvalue statistics of real and correlated Gaussian covariances match GOE RMT predictions (r-statistics ~0.536) with M=50000 samples.
- Effective convergence (RMT onset) occurs at sample sizes M_crit roughly on the order of the feature dimension d, much smaller than typical dataset sizes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Gaussian data with long-range feature correlations instead of uncorrelated Gaussian data | Gaussian dataset covariance model · Natural image datasets | Eigenvalue bulk power-law behavior: α ∼ 1/4 to α = −1 | Helped |
| Increase normally distributed noise corruption of FMNIST (ablation) | FMNIST · FMNIST covariance bulk | Power-law exponent α: α ∼ 1/4 to α = −1 | Hurt |
| Use CGD covariance with the corresponding real-world dataset's scaling exponent | CGD · FMNIST, CIFAR10, and ImageNet | Eigenvalue bulk distribution: KL=0.001; KL=0.006; KL=0.01 | Helped |
| Increase the number of samples to approximately the feature dimension | Empirical covariance matrix · CIFAR10, FMNIST, MNIST, and Gaussian datasets | Convergence of local and global spectral statistics: Mcrit ∼ d | Helped |
| Use strongly correlated data instead of uncorrelated data | Dataset covariance · Real and Gaussian datasets | Shannon entropy | Hurt |
| Use strongly correlated data instead of uncorrelated data | Dataset covariance · Real and Gaussian datasets | Samples required to reach distribution entropy | Helped |
Surprising findings
- Shannon entropy of covariance spectra is substantially lower for strongly correlated datasets, which also need fewer samples to reach the distribution entropy.
Limitations
- The origin of the observed chaotic spectral statistics (intrinsic correlations vs. sampling noise) remains unclear.
- Analysis restricted to Gram/covariance matrices, discarding spatial relationships; only image datasets studied.
- Relies on large d and M limits of RMT; finite-size deviations appear for higher-order r(n) statistics.
Experimental setup
Datasets: MNIST FMNIST CIFAR10 Tiny-ImageNet CelebA ImageNet
Benchmarks: MNIST FMNIST CIFAR10 Tiny-ImageNet CelebA ImageNet
Metrics: power-law exponent alpha KL divergence r-statistics level spacing distribution spectral form factor Shannon entropy convergence distances delta(M), Delta(M), epsilon(M)
Baselines: uncorrelated Gaussian data (UGD) GOE / Poisson RMT predictions
Builds on: Maloney et al. 2022 (solvable model of neural scaling laws) Wishart / Marchenko-Pastur law Oganesyan and Huse r-statistics Toeplitz matrices
Closest papers by meaning
- Models of Heavy-Tailed Mechanistic Universality (ICML 2025 · similarity 0.87)
- Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectra (ICLR 2025 · similarity 0.86)
- Asymptotics of Learning with Deep Structured (Random) Features (ICML 2024 · similarity 0.84)
- Small Singular Values Matter: A Random Matrix Analysis of Transformer Models (NeurIPS 2025 · similarity 0.84)
- Learning from higher-order correlations, efficiently: hypothesis tests, random features, and neural networks (NeurIPS 2024 · similarity 0.84)
- Super Consistency of Neural Network Landscapes and Learning Rate Transfer (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.