Research map / Probabilistic and scientific modeling / Variational and generative sampling
Spherical Boltzmann machines: a solvable theory of learning and generation in energy-based models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes a solvable spherical Boltzmann machine (SBM) trained on data in the high-dimensional limit, deriving its equilibrium phase diagram and Langevin training dynamics via DMFT, and predicts generative phenomena (temperature tuning, double descent, tempered posterior, out-of-equilibrium bias) that are checked numerically in standard architectures.
Problem
Energy-based models' learning and generative properties (partition function intractability, sampling temperature, regularization, posterior temperature, non-equilibrium training) remain poorly understood theoretically.
Approach
Treats the spherical Boltzmann machine as a solvable EBM: combines finite-rank spherical HCIZ asymptotics and a Coulomb-gas analysis of eigenvalues to get a closed-form equilibrium phase diagram, and uses DMFT on persistent-chain Langevin dynamics to obtain training dynamics and closed-form teacher-student generative metrics.
Key results
- Equilibrium phase diagram for SBM derived in closed form, with phases of aligned outliers, condensation, and coalesced modes; verified against finite-N simulations.
- Reverse KL double descent in regularization gamma occurs only for posterior temperature eta above a threshold eta_DD(omega*), per the closed-form analysis.
- Numerical experiments in Potts Boltzmann machines, normalizing flows, RBMs, and Bayesian GANs reproduce temperature tuning, double descent, and warm/cold posterior effects qualitatively.
Surprising findings
- Optimal posterior temperature can be below or above 1 (warm vs cold posterior effects) even in an exactly solvable EBM.
- Out-of-equilibrium persistent-chain training with finite sampling rate can yield a biased model whose KL first dips near equilibrium then increases.
Limitations
- The SBM is a minimal solvable model rather than a realistic generative architecture.
- Analysis assumes effectively low-dimensional data.
- Derivations are heuristic (saddle-point, MSR/DMFT closure) rather than rigorous proofs.
Experimental setup
Datasets: Pfam protein sequences financial dataset 2D Ising samples Curie-Weiss model samples synthetic datasets teacher-generated samples from rank-one SBM
Metrics: reverse KL divergence forward KL divergence cross-entropy top eigenvalue lambda_1 data overlap u_1
Baselines: Potts Boltzmann machines Bayesian GANs normalizing flow on sphere Gaussian-Bernoulli RBM autoregressive belief network
Builds on: Boltzmann machine learning with persistent contrastive divergence Harish-Chandra-Itzykson-Zuber integral Martin-Siggia-Rose path integral Bingham distribution
Closest papers by meaning
- Cascade of phase transitions in the training of energy-based models (NeurIPS 2024 · similarity 0.91)
- BoltzNCE: Learning likelihoods for Boltzmann Generation with Stochastic Interpolants and Noise Contrastive Estimation (NeurIPS 2025 · similarity 0.88)
- Fast training and sampling of Restricted Boltzmann Machines (ICLR 2025 · similarity 0.88)
- Learning with Restricted Boltzmann Machines: Asymptotics of AMP and GD in High Dimensions (NeurIPS 2025 · similarity 0.87)
- Energy Matching: Unifying Flow Matching and Energy-Based Models for Generative Modeling (NeurIPS 2025 · similarity 0.86)
- Scalable Equilibrium Sampling with Sequential Boltzmann Generators (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.