Research map / Architectures and efficiency / RNNs and normalization
Asymptotics of Learning with Deep Structured (Random) Features
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives a tight high-dimensional asymptotic characterization of ridge-regression test error for Lipschitz feature maps, applied to structured "Gaussian rainbow" deep random networks and shown to track trained lazy networks.
Problem
Characterizing the test error of learning the readout layer on features from deep non-linear networks with structured (correlated, non-unstructured) random weights, in the proportional high-dimensional regime.
Approach
Derives an anisotropic deterministic equivalent for ridge regression on general Lipschitz feature maps, expressed via the population covariances of the features, and gives closed-form linearized recursions for those covariances in Gaussian rainbow networks with correlated weights.
Key results
- Theorem gives test error E_gen^rmt matching numerical simulations at d=1000 for four-layer tanh Gaussian rainbow networks with correlated weights.
- For a deterministic three-layer ReLU network trained with Adam, the linearized theory predicts learning curves of readout retraining on MNIST with good agreement.
- The linearized covariance recursion yields an equivalent linear stochastic network sharing second moments and hence the same asymptotic test error.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the theoretical prediction with linearized feature covariances for a four-layer Gaussian rainbow network | Gaussian rainbow network readout ridge regression · Synthetic four-layer rainbow network | test error | No clear effect |
| Train the network feature map end-to-end before retraining the readout | Three-layer neural network · Synthetic data | test error | Helped |
| Use the theoretical prediction of the test error with estimated feature covariances | Trained three-layer neural network readout ridge regression · MNIST even/odd digit regression | test error | No clear effect |
| Use the theoretical prediction with linearized feature covariances | Trained three-layer neural network readout ridge regression · Regression on MNIST | test error | No clear effect |
| Use an initially randomly isotropic-initialized network feature map instead of the best ridge estimator | Three-layer neural network · Generalization across sample complexities | test error | Hurt |
Surprising findings
- Correlations across layers of weights do not enter the linearized population covariance formulas, though the test error still matches simulations.
Limitations
- Linearization rigorously proven only for two-layer case (L = L~ = 2); general depths only conjectured.
- Results apply to Gaussian rainbow / lazy-regime networks and may not capture feature learning in general trained networks.
- Error bounds worsen when parameter dimensions differ widely (comparable regime only).
Experimental setup
Datasets: MNIST synthetic Gaussian data matched to MNIST statistics
Benchmarks: MNIST
Metrics: test error (generalization error)
Baselines: unstructured deep random features linear regression ridge regression
Builds on: deep random features rainbow networks Hanson-Wright inequality linear pencils
Closest papers by meaning
- Deterministic equivalent and error universality of deep random features learning (ICML 2023 · similarity 0.89)
- High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representation (NeurIPS 2022 · similarity 0.88)
- Asymptotics of feature learning in two-layer networks after one gradient-step (ICML 2024 · similarity 0.88)
- Asymptotics of representation learning in finite Bayesian neural networks (NeurIPS 2021 · similarity 0.88)
- On the Universality of the Double Descent Peak in Ridgeless Regression (ICLR 2021 · similarity 0.87)
- Asymptotic Analysis of Two-Layer Neural Networks after One Gradient Step under Gaussian Mixtures Data with Structure (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.