Research map / Optimization / Implicit bias and generalization
Sliding Down the Stairs: How Correlated Latent Variables Accelerate Learning with Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that positive correlation between latent variables of different input cumulants lowers the information exponent along the higher-order cumulant direction, so online SGD on a single neuron or two-layer net learns the cumulant spike with ~d log^2 d samples instead of d^3.
Problem
Learning from higher-order input cumulants with SGD is computationally hard (tensor PCA needs ~d^{p-1} samples), which conflicts with the empirical importance and speed of HOCs in neural networks.
Approach
Introduces the mixed cumulant model (MCM), where a covariance spike and a cumulant spike share latent variables; positive correlation between these latents lowers the information exponent along the non-Gaussian direction, yielding nearly sharp sample-complexity thresholds for spherical SGD weak recovery.
Key results
- Covariance spike alone is learned with n ≳ d log^2 d samples, while cumulant spike alone needs n ≳ d^3 under online SGD.
- With independent latents, weak recovery of the cumulant spike is impossible for n ≲ d^3 at small learning rates; with positive latent correlation E[λν]>0, both spikes are weakly recovered with n ≳ d log^2 d (optimal step ~1/log d).
- Two-layer networks on the MCM show the predicted speed-up: with correlated latents the plateau disappears, separating learning of the cumulant from ~10^4 steps rather than ~10^6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Correlate the covariance and cumulant latent variables positively | MCM learning of the cumulant spike · MCM, spherical perceptron | sample complexity for weak recovery of v: n\gg d^3 (independent latents) -> n \gtrsim d\log^2 d (correlated latents) | Helped |
| Correlate covariance and cumulant latent variables | Two-layer network trained on MCM · MCM, two-layer neural network | steps to learn HOCs: \gtrsim 10^6 -> \gtrsim 10^{4} | Helped |
| Use correlated rather than independent latent variables | MCM learning of the cumulant spike · MCM, spherical perceptron with δ ≈ 1/d | sample complexity for weak recovery of v: n\ll d^3 (impossible with independent latents) -> n \gg d^2\log d (guaranteed with correlated latents) | Helped |
| Give covariance and cumulant spikes a finite overlap u·v = ρ, while keeping latent variables uncorrelated (ablation) | MCM learning of non-Gaussian correlations · MCM | speed of learning the non-Gaussian part | No clear effect |
| Use independent rather than positively correlated latent variables (ablation) | MCM learning of the cumulant spike · MCM, spherical perceptron | sample complexity for weak recovery of v: n \gtrsim d\log^2 d -> n\gtrsim d^3 | Hurt |
| Learn the cumulant spike alone rather than alongside a correlated covariance spike (ablation) | MCM learning of the cumulant spike · Cumulant-spike-only model versus correlated two-spike MCM | sample complexity for weak recovery of v: n\gg d\log^2 d -> n\gg d^3\log^2 d | Hurt |
| Remove the covariance spike, leaving only the cumulant spike (ablation) | MCM learning of the cumulant spike · Cumulant-spike-only model | sample complexity for weak recovery of v: n \gtrsim d\log^2 d (correlated two-spike model) -> n\gg d^3\log^2 d | Hurt |
| Use a learning rate δ ≈ 1/d instead of δ ≈ 1/log d (ablation) | Correlated two-spike MCM · MCM, spherical perceptron; weak recovery of v | sample complexity: n \gg d\log^2 d -> n \gg d^2\log d | Hurt |
| Condition on matching initialization signs for the two spike overlaps | Correlated two-spike MCM · MCM, spherical perceptron; weak recovery of v | weak recovery | Helped |
Surprising findings
- Correlating the covariance and cumulant spikes directionally (overlap u·v) does not speed up learning; only correlation between the latent variables does.
- Mismatched signs of initial overlaps with correlated latents can push both overlaps toward zero, slowing learning for a single neuron.
Limitations
- Analysis covers only the search phase of SGD for a single spherical perceptron; the descent phase and two-layer dynamics are not analysed rigorously.
- Single-neuron results require initialisation with matching overlap signs; wide two-layer networks avoid this in simulations but no proof is given.
- Positive results for independent-latent case beyond the horizon d log^2 d are not controlled.
Experimental setup
Datasets: CIFAR10
Benchmarks: CIFAR10
Metrics: test error test loss weak recovery overlap
Baselines: independent latent variables MCM Gaussian equivalent model
Builds on: tensor PCA information exponent spherical perceptron SGD analysis staircase phenomenon
Closest papers by meaning
- Learning from higher-order correlations, efficiently: hypothesis tests, random features, and neural networks (NeurIPS 2024 · similarity 0.92)
- Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectra (ICLR 2025 · similarity 0.84)
- From Information to Generative Exponent: Learning Rate Induces Phase Transitions in SGD (NeurIPS 2025 · similarity 0.84)
- Enhancing Neural Training via a Correlated Dynamics Model (ICLR 2024 · similarity 0.84)
- Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning (NeurIPS 2024 · similarity 0.84)
- Learning quadratic neural networks in high dimensions: SGD dynamics and scaling laws (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.