Research map / Probabilistic and scientific modeling / Variational and generative sampling
Brain-like Variational Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives a family of iterative VAEs from free-energy natural-gradient dynamics (FOND), including i-P-VAE, a recurrent spiking model whose membrane-potential dynamics perform online variational inference.
Problem
How variational inference (ELBO maximization / free energy minimization) can be implemented as brain-like neural dynamics, and how iterative inference compares with amortized inference on sparsity, reconstruction, and biological plausibility.
Approach
FOND derives inference dynamics from three prescriptions (natural gradients on free energy, online belief updating, iterative refinement); with Poisson posterior/prior and Gaussian likelihood this yields membrane-potential dynamics with explaining-away recurrence, leak, and emergent divisive normalization, producing integer spike counts.
Key results
- i-P-VAE reaches R^2 0.83 with 77% zeros at t=1000 inference steps, versus 58% zeros for i-G-ReLU-VAE and 0% for i-G-VAE (R^2 0.87).
- Iterative VAEs consistently outperform their amortized counterparts in Euclidean distance to the reconstruction-sparsity optimum, and i-P-VAE is statistically indistinguishable from LCA.
- i-P-VAE uses about 25x fewer parameters than amortized convolutional encoders while producing integer-valued spike counts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Poisson rather than Gaussian latents (ablation) | Iterative VAE · van Hateren 16 × 16 patches | sparsity (proportion of zeros): 0% -> 77% | Helped |
| Use Gaussian rather than Poisson latents (ablation) | Iterative VAE · van Hateren 16 × 16 patches | R²: 0.83 -> 0.87 | Helped |
| Use Gaussian latents with a post-sampling ReLU rather than Poisson latents (ablation) | Iterative VAE · van Hateren 16 × 16 patches | sparsity (proportion of zeros): 77% -> 58% | Hurt |
| Use Gaussian latents with a post-sampling ReLU rather than Poisson latents (ablation) | Iterative VAE · van Hateren 16 × 16 patches | R²: 0.83 -> 0.82 | Hurt |
| Use iterative rather than amortized inference | VAE models · van Hateren 16 × 16 patches | distance from the reconstruction-sparsity optimum | Helped |
| Use iterative inference rather than amortized inference | VAE models · van Hateren 16 × 16 patches | inference speed | No clear effect |
| Increase training inference steps from T_train=8 to T_train=16 or 32 (ablation) | iP-VAE · van Hateren 16 × 16 patches | reconstruction-sparsity trade-off | Helped |
| Increase β (ablation) | iP-VAE · van Hateren 16 × 16 patches | sparsity | Helped |
| Use iP-VAE rather than LCA | Sparse coding model comparison · van Hateren 16 × 16 patches | reconstruction-sparsity performance | No clear effect |
| Use iterative Poisson inference rather than the amortized P-VAE | Poisson VAE · MNIST | OOD reconstruction and classification accuracy | Helped |
Surprising findings
- The i-G-VAE achieves the best R^2 but has no sparsity, while the Poisson variant trades a little reconstruction for substantial sparsity.
- Amortized PC-style and hybrid baselines are outperformed by iterative inference even though the amortized encoders have far more parameters.
Limitations
- Main results use linear decoders on small 16x16 patches; nonlinear decoder and color-image results are deferred to the appendix.
- Iterative inference requires many test-time steps (T_test=1000) to converge, increasing inference cost.
- Gradient norm plateaus rather than vanishing, attributed to stochastic gradient variance.
Experimental setup
Datasets: van Hateren natural image patches MNIST CelebA CIFAR-10 tiny ImageNet
Benchmarks: van Hateren whitened patches MNIST
Metrics: R^2 reconstruction proportion of zeros (sparsity) gradient norm distance to optimum classification accuracy
Baselines: Gaussian VAE Poisson VAE Gaussian-ReLU VAE predictive coding incremental predictive coding locally competitive algorithm hybrid iterative-amortized VAE
Builds on: Bayesian learning rule Poisson VAE locally competitive algorithm predictive coding variational autoencoder
Closest papers by meaning
- Globally Convergent Variational Inference (NeurIPS 2024 · similarity 0.90)
- Variational methods for simulation-based inference (ICLR 2022 · similarity 0.89)
- Poisson Variational Autoencoder (NeurIPS 2024 · similarity 0.88)
- The Thermodynamic Variational Objective (NeurIPS 2019 · similarity 0.88)
- Backpropagation at the Infinitesimal Inference Limit of Energy-Based Models: Unifying Predictive Coding, Equilibrium Propagation, and Contrastive Hebbian Learning (ICLR 2023 · similarity 0.88)
- SING: SDE Inference via Natural Gradients (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.