Research map / Probabilistic and scientific modeling / Variational and generative sampling
Exponential-Family Harmoniums with Neural Sufficient Statistics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends exponential-family harmoniums by letting hidden-unit sufficient statistics of the visible data be deep neural networks, sampling with Langevin-within-Gibbs, and generating CIFAR-10 and CelebA-HQ samples competitive with NCSN.
Problem
Restricted Boltzmann machines and exponential-family harmoniums cannot use convolutional or deep feature extractors on the emission side and have struggled to generate complex images like CIFAR-10.
Approach
Let the emission sufficient statistics be an arbitrary deep neural network (keeping Bernoulli posteriors), which breaks standard block Gibbs sampling; sampling is restored by Langevin dynamics within Gibbs, with persistent Langevin chains inspired by GRBM training.
Key results
- NN-EFH achieves CIFAR-10 FID 32.1, versus 42.3 for the EBM baseline and 164 for the GRBM under Gibbs-Langevin, and competitive with NCSN (25.2).
- On CelebA, NN-EFH FID is 11.3 versus 22.7 for the EBM base, comparable to DCGAN (12.5).
- Linear probe accuracy on CIFAR-10 using sufficient statistics U(x) is 53%, versus 29% for the EBM penultimate layer.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add neural sufficient statistics and latent variables with Gibbs-Langevin sampling instead of the comparable EBM | NN-EFH · CelebA | FID: 22.7 -> 11.3 | Helped |
| Add neural sufficient statistics and latent variables with Gibbs-Langevin sampling instead of the comparable EBM | NN-EFH · CIFAR-10 | FID: 42.3 -> 32.1 | Helped |
| Use NCSN instead of NN-EFH | CIFAR-10 · CIFAR-10 | FID: 32.1 -> 25.2 | Helped |
| Use NN-EFH instead of GRBM | CIFAR-10 · CIFAR-10 | FID: 164 -> 32.1 | Helped |
| Use NN-EFH sufficient statistics U(x) instead of EBM penultimate-layer features | Linear classifier · CIFAR-10 | accuracy: 29% -> 53% | Helped |
| Use NN-EFH sufficient statistics U(x) instead of EBM penultimate-layer features | Linear classifier · FMNIST | accuracy: 35% -> 71% | Helped |
| Use NN-EFH natural parameters η(x) instead of EBM penultimate-layer features | Linear classifier · CIFAR-10 | accuracy: 29% -> 47% | Helped |
| Use NN-EFH natural parameters η(x) instead of EBM penultimate-layer features | Linear classifier · FMNIST | accuracy: 35% -> 62% | Helped |
| Initialize the test-time Gibbs sampler at data rather than noise (ablation) | NN-EFH generation · Image generation | sample quality | Helped |
| Reinitialize the Langevin sampler from noise at each Gibbs iteration instead of persisting its state (ablation) | NN-EFH training · Image generation | sample quality | Hurt |
| Use Gibbs-Langevin instead of Langevin dynamics alone (ablation) | Sampling dynamics · Model-space exploration | distance traveled: approximately 10 times greater | Helped |
Surprising findings
- The GRBM trained with the same Gibbs-Langevin procedure fails to generate realistic CIFAR-10 samples (FID 164).
- Metropolis adjustment was omitted because acceptance rates were very small, and the method works in practice without it.
Limitations
- Sampling requires many Langevin steps per Gibbs step, so generation is slow compared with feed-forward generators.
- Metropolis adjustment is omitted, so the chain is only approximately sampling from the stationary distribution.
- Only modest network sizes were used and results are reported only on images up to 64x64.
Experimental setup
Datasets: MNIST Fashion-MNIST Oxford Flowers CelebA CelebA-HQ CIFAR-10
Benchmarks: CIFAR-10 FID CelebA-HQ FID CelebA FID CIFAR-10 linear probe accuracy FMNIST linear probe accuracy
Metrics: Frechet Inception distance linear evaluation accuracy nearest-neighbor distance
Baselines: GRBM (Gibbs-Langevin) EBM with same architecture NCSN DCGAN VAE Glow JEM IGEBM FlowCE NT-EBM
Builds on: restricted Boltzmann machine exponential-family harmonium Gaussian-Bernoulli RBM Langevin-within-Gibbs energy-based model short-run Langevin
Closest papers by meaning
- eXponential FAmily Dynamical Systems (XFADS): Large-scale nonlinear Gaussian state-space modeling (NeurIPS 2024 · similarity 0.83)
- Fast training and sampling of Restricted Boltzmann Machines (ICLR 2025 · similarity 0.83)
- ELoRA: Low-Rank Adaptation for Equivariant GNNs (ICML 2025 · similarity 0.82)
- The Importance of Being Scalable: Improving the Speed and Accuracy of Neural Network Interatomic Potentials Across Chemical Domains (NeurIPS 2024 · similarity 0.82)
- Spherical Boltzmann machines: a solvable theory of learning and generation in energy-based models (NeurIPS 2026 · similarity 0.82)
- Cascade of phase transitions in the training of energy-based models (NeurIPS 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.