Research map / Optimization / Implicit bias and generalization
Learning with Restricted Boltzmann Machines: Asymptotics of AMP and GD in High Dimensions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that RBM likelihood training in the high-dimensional limit (fixed hidden units) reduces to an unsupervised multi-index model, and uses AMP and DMFT to give rigorous asymptotics of RBM learning on spiked covariance data, recovering the BBP weak recovery threshold.
Problem
Rigorous understanding of how RBMs learn via maximum likelihood from data is lacking; prior analyses cover only SVD-like Gaussian cases or replica/Bayesian posterior sampling that does not match practical likelihood training.
Approach
In the limit of large visible dimension and finitely many hidden units, the RBM log-likelihood is asymptotically equivalent to an unsupervised multi-index model with a non-separable regularizer, which lets AMP (AMP-RBM) and its state evolution, plus dynamical mean-field theory for gradient descent, characterize training on spiked covariance data.
Key results
- AMP-RBM weak recovery threshold matches the BBP transition lambda_max = alpha^{-1/4}, i.e., alpha*lambda^4 <= 1.
- AMP-RBM and GD state evolutions closely track the empirical overlaps for r=k=2, n=8000, d=4000 with Rademacher prior.
- RBM likelihood is asymptotically equivalent (per dimension) to a multi-index model with non-separable regularization as d grows with fixed k.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train the RBM by likelihood maximization | RBM on spiked covariance data · Spiked covariance model | weak recovery threshold | Helped |
Surprising findings
- Replica/Bayesian teacher-student analyses predict a spin-glass phase at zero temperature, whereas likelihood-trained RBMs always match the BBP weak recovery threshold.
Limitations
- Analysis requires large dimension with fixed number of hidden units and n/d fixed ratio.
- Global optimality proven only for k=r=1; general case is conjectured.
- Zero-mean visible unit assumption; non-zero mean case left as future work.
Experimental setup
Datasets: spiked covariance synthetic data
Benchmarks: spiked covariance synthetic data
Metrics: overlap with teacher weights weak recovery threshold
Baselines: gradient descent Bayes-optimal estimator SVD
Builds on: approximate message passing dynamical mean-field theory spiked covariance model BBP transition
Closest papers by meaning
- Learning Restricted Boltzmann Machines with Sparse Latent Variables (NeurIPS 2020 · similarity 0.92)
- From Boltzmann Machines to Neural Networks and Back Again (NeurIPS 2020 · similarity 0.91)
- Fast training and sampling of Restricted Boltzmann Machines (ICLR 2025 · similarity 0.90)
- Cascade of phase transitions in the training of energy-based models (NeurIPS 2024 · similarity 0.87)
- Spherical Boltzmann machines: a solvable theory of learning and generation in energy-based models (NeurIPS 2026 · similarity 0.87)
- BoltzNCE: Learning likelihoods for Boltzmann Generation with Stochastic Interpolants and Noise Contrastive Estimation (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.