Research map / Probabilistic and scientific modeling / Variational and generative sampling
Sample as you Infer: Predictive Coding with Langevin Dynamics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Langevin Predictive Coding (LPC), which injects Gaussian noise into predictive coding inference to get unadjusted Langevin sampling of latent states, trains an amortized encoder warm-start with forward/reverse/Jeffreys KL objectives, and adds diagonal Adam-style preconditioning to train deep generative models compared against VAEs.
Problem
Predictive coding (PC) inference yields only MAP point estimates of latent states, which restricts model quality, and PC has not been shown to train deep generative models competitively with VAEs.
Approach
Noise-injected PC inference is an unadjusted overdamped Langevin sampler; its samples give a Monte Carlo ELBO gradient for model parameters, an amortized encoder warm-starts the chain (trained with forward KL, reverse KL, or Jeffreys divergence), and a diagonal Adam-inspired preconditioner improves robustness to step size.
Key results
- LPC (15 epochs) reaches FID 39.64 on SVHN versus 44.76 for VAE at 50 epochs, and FID 113.29 on CIFAR-10 versus 145.87 for VAE at 50 epochs.
- On CelebA, VAE at 50 epochs reaches FID 82.09 versus 97.49 for LPC at 15 epochs, so LPC does not win on every dataset.
- End-to-end training wall-clock is about 7x (reverse KL) and 11x (Jeffreys) slower than the VAE on an Nvidia 4090.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LPC instead of VAE (50 Epochs) | Generative model · CelebA | FID: 82.09 -> 97.49 | Hurt |
| Use LPC instead of VAE (50 Epochs) | Generative model · SVHN | FID: 44.76 -> 39.64 | Helped |
| Use LPC instead of VAE (50 Epochs) | Generative model · CIFAR10 | FID: 145.87 -> 113.29 | Helped |
| Use LPC instead of VAE (50 Epochs) | Generative model · CelebA | Density: 0.16 -> 0.54 | Helped |
| Use LPC instead of VAE (50 Epochs) | Generative model · SVHN | Density: 0.65 -> 0.33 | Hurt |
| Use LPC instead of VAE (50 Epochs) | Generative model · CIFAR10 | Density: 0.14 -> 0.63 | Helped |
| Use LPC instead of VAE (50 Epochs) | Generative model · CelebA | Coverage: 0.12 -> 0.13 | Helped |
| Use LPC instead of VAE (50 Epochs) | Generative model · SVHN | Coverage: 0.48 -> 0.42 | Hurt |
| Use LPC instead of VAE (50 Epochs) | Generative model · CIFAR10 | Coverage: 0.06 -> 0.13 | Helped |
| Train amortized warm-starts with reverse KL instead of no warm-start (ablation) | LPC · CIFAR-10, SVHN, CelebA | FID | Helped |
| Train amortized warm-starts with Jeffrey's divergence instead of no warm-start (ablation) | LPC · CIFAR-10, SVHN, CelebA | FID | Helped |
| Train amortized warm-starts with forward KL instead of no warm-start (ablation) | LPC · CIFAR-10, SVHN, CelebA | FID | Hurt |
Surprising findings
- Forward KL warm-start training was unstable, with exploding gradients and poor initialization as training progressed, so its FID is reported at 1 epoch only.
- At the smallest Langevin step size (1e-3), non-preconditioned models matched or outperformed preconditioned ones.
Limitations
- Sequential Langevin iterations per SGD step make training substantially slower in wall-clock time.
- Forward KL objective is unstable, with exploding gradients at later epochs.
- Results on CelebA do not favor LPC over the 50-epoch VAE on FID.
Experimental setup
Datasets: CIFAR-10 SVHN CelebA 64x64
Benchmarks: CIFAR-10 SVHN CelebA 64x64
Metrics: FID density coverage
Baselines: VAE VAE trained for 50 epochs LPC without warm-start (prior initialization) LPC without preconditioning
Builds on: predictive coding unadjusted Langevin algorithm variational autoencoder Adam stochastic gradient Langevin dynamics
Closest papers by meaning
- Divide-and-Conquer Predictive Coding: a structured Bayesian inference algorithm (NeurIPS 2024 · similarity 0.88)
- The Expected Loss of Preconditioned Langevin Dynamics Reveals the Hessian Rank (AAAI 2024 · similarity 0.88)
- Preconditioned Langevin Dynamics with Score-based Generative Models for Infinite-Dimensional Linear Bayesian Inverse Problems (NeurIPS 2025 · similarity 0.88)
- Microcanonical Langevin Ensembles: Advancing the Sampling of Bayesian Neural Networks (ICLR 2025 · similarity 0.88)
- PID-controlled Langevin Dynamics for Faster Sampling on Generative Models (NeurIPS 2025 · similarity 0.87)
- Only Strict Saddles in the Energy Landscape of Predictive Coding Networks? (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.