AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

How to Center Deep Boltzmann Machines

Jan Melchior, Asja Fischer, Laurenz Wiskott

JMLR 2016 · Architectures and efficiency › RNNs and normalization · approach: Weight normalization and decay interplay · 40 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
center visible and hidden units (ddl_s) instead of normal RBMRBM trained with PCD-1 lr 0.01 · MNIST (16 hidden)max avg training log-likelihood: -193.60 -> -140.43Improved
center visible and hidden units (aab) instead of normal RBMRBM trained with CD-1 lr 0.1 · Bars & Stripes (4 hidden)max avg training log-likelihood: -66.53 -> -61.66Improved
center hidden units too (ddl_s) rather than visible only (d0)centered RBM, PCD-1 lr 0.01 · Bars & Stripes (4 hidden)max avg training log-likelihood: -61.03 -> -56.80Improved
use enhanced gradient offsets (aab) with PCD or PTcentered RBM · Bars & Stripes (4 hidden), PCD-1 lr 0.1max avg training log-likelihood: -59.05 (end of training -360.6)Hurt
use parallel tempering (PT) instead of CD/PCDddl_s, d0, 00 RBMs · Bars & Stripes (4 hidden)max avg training log-likelihood: -56.80 -> -52.21 (PCD-1 lr 0.01 vs PT10 lr 0.05, ddl_s)Improved
initialize visible bias to inverse sigmoid of data mean instead of zero (ablation)normal RBM, CD-1 lr 0.2 · Flipped Shifting Bar (4 hidden)max avg training log-likelihood: -28.14 -> -22.08Improved
initialize visible bias to inverse sigmoid of data mean instead of zero (ablation)normal RBM, PCD-1 lr 0.01 · MNIST (16 hidden)max avg training log-likelihood: -193.60 -> -144.28Improved
reparameterize before instead of after the gradient update (ablation)centered RBM (ddb_s vs ddl_s) · MNIST (16 hidden)max avg training log-likelihood: -144.12 vs -144.10 (PCD-1 lr 0.1)No clear effect
use exact offsets/gradient instead of sampled estimates (ablation)aab with PT · Bars & Stripes (4 hidden)divergence: divergence removedImproved
use an exponentially moving average (sliding factor 0.01) for offsets (ablation)centered RBM using model mean offsets · Bars & Stripes (4 hidden)divergence / max LL: divergence removed; maximum LL unchangedMixed
fix offsets to 0.5 instead of data expectations (ablation)centered RBM · MNISTmax avg training log-likelihood: worse than normal RBM on version with fewer onesHurt
center the RBM (aab, ddb_s, ddl_s, d0) vs 00 with 500 hidden unitscentered RBM · MNIST and Caltech (500 hidden, AIS test LL)test log-likelihood: higher LL; no divergence up to 120000 updates vs 00 and d0 diverging after 30000Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy