Research map / Probabilistic and scientific modeling / Molecules and proteins

Cascade of phase transitions in the training of energy-based models

NeurIPS 2024 · Probabilistic and scientific modeling › Molecules and proteins · approach: Sparse concepts and hypothesis testing · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analytically and numerically shows that RBM weight-matrix SVD modes are learned through a cascade of mean-field-like second-order phase transitions, with finite-size scaling confirming the first transition's universality class.

Problem

How the feature encoding process of energy-based models (RBMs) unfolds during training, i.e., how the learned probability distribution acquires modes/features during gradient-based learning.

Approach

Shows that RBM training proceeds through a sequence of second-order (mean-field, Curie-Weiss/Mattis-like) phase transitions in which each singular mode of the weight matrix grows exponentially and triggers a split of the learned distribution into more modes; the first transition is tested via susceptibility divergence and mean-field finite-size scaling.

Key results

• On MNIST the susceptibility of the first mode follows a Curie-Weiss form 4/(w_c^2 - w^2) with fitted critical point w_1c = 4.45 and no adjustable prefactor.
• Mean-field finite-size scaling with gamma=1, nu=1/2, d_u=4 collapses susceptibility data near the critical region for the first transition across system sizes.
• Analytically for the BG-RBM with one hidden node on Mattis data, the susceptibility diverges with exponent gamma=1 at N_v^{-1} sum w_k^2 ~ 1, and weights converge to norm sqrt(beta).

Measured effects reported in the paper

Surprising findings

• The alignment of singular vectors with PCA directions is only temporary; the RBM later finds different patterns as training progresses.
• MCMC relaxation times diverge at each phase transition (critical slowing down), which hampers sampling during training.

Limitations

• Analytical results rely on simplified setups (single or few hidden nodes, Mattis/Hopfield data, zero biases), and the PCA alignment is only temporary.
• Biases and non-equilibrium MCMC are not fully covered by the theory; the paper states a limitation of current theoretical approaches.

Experimental setup

Datasets: MNIST CelebA Human Genome Dataset (HGD) Mattis model synthetic data Hopfield model synthetic data

Benchmarks: MNIST CelebA HGD

Metrics: singular values susceptibility magnetization variance finite-size scaling collapse MCMC relaxation time

Builds on: Restricted Boltzmann Machine Mattis model Hopfield model Persistent Contrastive Divergence

Closest papers by meaning

• Fast training and sampling of Restricted Boltzmann Machines (ICLR 2025 · similarity 0.91)
• Spherical Boltzmann machines: a solvable theory of learning and generation in energy-based models (NeurIPS 2026 · similarity 0.91)
• From Density Matrices to Phase Transitions in Deep Learning: Spectral Early Warnings and Interpretability (NeurIPS 2026 · similarity 0.88)
• Learning with Restricted Boltzmann Machines: Asymptotics of AMP and GD in High Dimensions (NeurIPS 2025 · similarity 0.87)
• Energy Loss Functions for Physical Systems (NeurIPS 2025 · similarity 0.86)
• BoltzNCE: Learning likelihoods for Boltzmann Generation with Stochastic Interpolants and Noise Contrastive Estimation (NeurIPS 2025 · similarity 0.86)