Research map / Architectures and efficiency / RNNs and normalization
How to Center Deep Boltzmann Machines
Jan Melchior, Asja Fischer, Laurenz Wiskott
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| center visible and hidden units (ddl_s) instead of normal RBM | RBM trained with PCD-1 lr 0.01 · MNIST (16 hidden) | max avg training log-likelihood: -193.60 -> -140.43 | Improved |
| center visible and hidden units (aab) instead of normal RBM | RBM trained with CD-1 lr 0.1 · Bars & Stripes (4 hidden) | max avg training log-likelihood: -66.53 -> -61.66 | Improved |
| center hidden units too (ddl_s) rather than visible only (d0) | centered RBM, PCD-1 lr 0.01 · Bars & Stripes (4 hidden) | max avg training log-likelihood: -61.03 -> -56.80 | Improved |
| use enhanced gradient offsets (aab) with PCD or PT | centered RBM · Bars & Stripes (4 hidden), PCD-1 lr 0.1 | max avg training log-likelihood: -59.05 (end of training -360.6) | Hurt |
| use parallel tempering (PT) instead of CD/PCD | ddl_s, d0, 00 RBMs · Bars & Stripes (4 hidden) | max avg training log-likelihood: -56.80 -> -52.21 (PCD-1 lr 0.01 vs PT10 lr 0.05, ddl_s) | Improved |
| initialize visible bias to inverse sigmoid of data mean instead of zero (ablation) | normal RBM, CD-1 lr 0.2 · Flipped Shifting Bar (4 hidden) | max avg training log-likelihood: -28.14 -> -22.08 | Improved |
| initialize visible bias to inverse sigmoid of data mean instead of zero (ablation) | normal RBM, PCD-1 lr 0.01 · MNIST (16 hidden) | max avg training log-likelihood: -193.60 -> -144.28 | Improved |
| reparameterize before instead of after the gradient update (ablation) | centered RBM (ddb_s vs ddl_s) · MNIST (16 hidden) | max avg training log-likelihood: -144.12 vs -144.10 (PCD-1 lr 0.1) | No clear effect |
| use exact offsets/gradient instead of sampled estimates (ablation) | aab with PT · Bars & Stripes (4 hidden) | divergence: divergence removed | Improved |
| use an exponentially moving average (sliding factor 0.01) for offsets (ablation) | centered RBM using model mean offsets · Bars & Stripes (4 hidden) | divergence / max LL: divergence removed; maximum LL unchanged | Mixed |
| fix offsets to 0.5 instead of data expectations (ablation) | centered RBM · MNIST | max avg training log-likelihood: worse than normal RBM on version with fewer ones | Hurt |
| center the RBM (aab, ddb_s, ddl_s, d0) vs 00 with 500 hidden units | centered RBM · MNIST and Caltech (500 hidden, AIS test LL) | test log-likelihood: higher LL; no divergence up to 120000 updates vs 00 and d0 diverging after 30000 | Improved |
Closest papers by meaning
- End-to-end Training of Deep Boltzmann Machines by Unbiased Contrastive Divergence with Local Mode Initialization (ICML 2023 · similarity 0.84)
- Learning with Restricted Boltzmann Machines: Asymptotics of AMP and GD in High Dimensions (NeurIPS 2025 · similarity 0.84)
- Fast training and sampling of Restricted Boltzmann Machines (ICLR 2025 · similarity 0.84)
- Spherical Boltzmann machines: a solvable theory of learning and generation in energy-based models (NeurIPS 2026 · similarity 0.82)
- Simplicity Bias and Optimization Threshold in Two-Layer ReLU Networks (ICML 2025 · similarity 0.81)
- BoltzNCE: Learning likelihoods for Boltzmann Generation with Stochastic Interpolants and Noise Contrastive Estimation (NeurIPS 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
