Research map / Probabilistic and scientific modeling / Variational and generative sampling
Fast training and sampling of Restricted Boltzmann Machines
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes trajectory-based annealing for RBM log-likelihood estimation, a parallel trajectory tempering sampler using saved training checkpoints, and low-rank convex pretraining to handle clustered datasets.
Problem
Training, evaluating (log-likelihood), and sampling equilibrium RBMs is hampered by slow MCMC mixing on highly structured, multimodal datasets, where critical slowdown and first-order transitions arise.
Approach
Uses the training trajectory itself as an annealing path: online trajectory AIS updates the partition function at each parameter update, Parallel Trajectory Tempering swaps configurations between saved checkpoints instead of temperatures, and low-rank RBMs obtained by convex optimization pretrain the model to encode principal components.
Key results
- Online Tr-AIS log-likelihood estimates are more accurate than temperature AIS in controlled experiments with exact partition functions (20 hidden nodes).
- PTT yields substantially more mode jumps than AGS, PT, and Stacked Tempering across datasets, with the authors reporting it consistently outperforms all other algorithms.
- Pretraining with a low-rank RBM improves sample balance and test log-likelihood on clustered datasets (HGD, MNIST01, Mickey, Ising), but offers little benefit on CelebA and full MNIST.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add low-rank RBM pretraining before PCD (ablation) | RBM training · Highly structured datasets | test log-likelihood | Helped |
| Add low-rank RBM pretraining before PCD (ablation) | RBM training · CelebA and MNIST | sample quality | No clear effect |
| Add low-rank RBM pretraining before PCD (ablation) | RBM training · CelebA | training time (s): $119.2\pm 0.8$ -> $3134.0\pm 0.6$ | Hurt |
| Add low-rank RBM pretraining before PCD (ablation) | RBM training · HGD | training time (s): $106.2 \pm 0.3$ -> $266 \pm 3$ | Hurt |
| Add low-rank RBM pretraining before PCD (ablation) | RBM training · Ising | training time (s): $62 \pm 1$ -> $104\pm 2$ | Hurt |
| Add low-rank RBM pretraining before PCD (ablation) | RBM training · Mickey | training time (s): $118.3\pm 3$ -> $246.0\pm 0.7$ | Hurt |
| Add low-rank RBM pretraining before PCD (ablation) | RBM training · MNIST-01 | training time (s): $100.6\pm 0.3$ -> $258.0\pm 0.9$ | Hurt |
| Add low-rank RBM pretraining before PCD (ablation) | RBM training · MNIST | training time (s): $105.7\pm 0.2$ -> $233.9\pm 0.2$ | Hurt |
| Use PTT instead of AGS, PT, PTref, or Stacked Tempering | RBM sampling · MNIST01, HGD, and Ising 2D | number of mode jumps | Helped |
| Use training-trajectory annealing instead of temperature AIS | RBM log-likelihood estimation · HGD and controlled datasets | log-likelihood estimation error | Helped |
| Use PTT for offline log-likelihood estimation | RBM log-likelihood estimation · HGD and controlled datasets | log-likelihood estimation error | Helped |
| Use temperature-based parallel tempering rather than PTT | RBM sampling · HGD | recovery of minority cluster | Hurt |
Limitations
- Low-rank pretraining is limited to intrinsic dimensions d <= 4 due to discretization memory constraints.
- Pretraining primarily mitigates initialization effects; mixing issues may reappear later in training, and benefits diminish for extended training or less clustered data.
- Log-likelihood accuracy is validated against exact partition functions only for small models with up to 20 hidden nodes.
Experimental setup
Datasets: MNIST-01 Mickey Human Genome Dataset (HGD) Ising 2D CelebA MNIST
Benchmarks: MNIST-01 Mickey HGD Ising 2D CelebA MNIST
Metrics: log-likelihood estimation error vs exact partition function mode jump count autocorrelation time PCA projection agreement test log-likelihood
Baselines: persistent contrastive divergence annealed importance sampling parallel tempering Stacked Tempering alternating Gibbs sampling population annealing Jarzynski reweighting
Builds on: Restricted Coulomb Machine low-rank RBM persistent contrastive divergence parallel tempering Stacked Tempering annealed importance sampling
Closest papers by meaning
- Cascade of phase transitions in the training of energy-based models (NeurIPS 2024 · similarity 0.91)
- Learning with Restricted Boltzmann Machines: Asymptotics of AMP and GD in High Dimensions (NeurIPS 2025 · similarity 0.90)
- Spherical Boltzmann machines: a solvable theory of learning and generation in energy-based models (NeurIPS 2026 · similarity 0.88)
- BoltzNCE: Learning likelihoods for Boltzmann Generation with Stochastic Interpolants and Noise Contrastive Estimation (NeurIPS 2025 · similarity 0.87)
- Temperature-Annealed Boltzmann Generators (ICML 2025 · similarity 0.87)
- Scalable Equilibrium Sampling with Sequential Boltzmann Generators (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.