Research map / Probabilistic and scientific modeling / Variational and generative sampling
Poisson Variational Autoencoder
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the Poisson VAE (P-VAE), whose latents are discrete Poisson spike counts sampled with a relaxed Poisson reparameterization, yielding a rate-penalty ELBO that reduces to amortized sparse coding with a linear decoder.
Problem
Standard VAEs use continuous Gaussian latents, which diverge from the discrete, positive, spike-count nature of biological neurons; the paper asks whether Poisson latents can give a brain-like VAE that links to sparse coding and predictive coding.
Approach
Parameterize the posterior as Poisson with rates equal to prior rates times encoder residual rates (predictive-coding style), derive a KL term that penalizes firing rates, and sample differentiable Poisson counts via a sigmoid-relaxed exponential-waiting-time reparameterization with annealed temperature.
Key results
- P-VAE reaches ~82% KNN accuracy with only N=200 labels, which Gaussian VAE needs N=1000 for, i.e. about 5x sample efficiency.
- P-VAE keeps ~2% dead neurons on natural image datasets versus ~80% for Gaussian and Laplace VAEs.
- Conv-encoder P-VAE reaches lifetime sparsity s≈0.94 with MSE comparable to LCA, but a large amortization gap remains vs LCA sparse coding.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Poisson latents instead of Gaussian latents | VAE representations · MNIST validation, KNN with 200 labeled samples | accuracy: 0.673 ± .003 -> 0.815 ± .002 | Helped |
| use Poisson latents instead of Gaussian latents | VAE representations · MNIST validation, shattering dimension, K=10 | shattering dimension: 0.758 ± .007 -> 0.797 ± .009 | Helped |
| use Poisson latents instead of continuous Gaussian or Laplace latents | VAE · van Hateren and CIFAR16x16 | proportion of active neurons: ∼ 80% dead neurons -> ∼ 2% dead neurons | Helped |
| use a Monte Carlo Poisson reparameterization estimator instead of exact gradients | P-VAE with linear decoder · van Hateren, linear encoder and decoder | percent drop in validation loss: 0.6 ± .5 -> 0.0 ± .1 | No clear effect |
| use straight-through gradients instead of exact gradients (ablation) | P-VAE with linear decoder · van Hateren, linear encoder and decoder | percent drop in validation loss: 0.6 ± .5 -> 7.3 ± .1 | Hurt |
| use hard-forward surrogate gradients instead of relaxed Poisson training (ablation) | P-VAE · P-VAE training | performance | Hurt |
| reduce the final temperature from 0.1 to 0.05 (ablation) | P-VAE · P-VAE training | performance | Helped |
| vary the temperature annealing schedule between linear and exponential (ablation) | P-VAE · van Hateren | performance | No clear effect |
| apply ReLU to sampled Gaussian latents (ablation) | Gaussian VAE · MNIST validation, KNN with 200 labeled samples | accuracy: 0.673 ± .003 -> 0.694 ± .003 | Helped |
| apply exp to sampled Gaussian latents (ablation) | Gaussian VAE · MNIST validation, KNN with 200 labeled samples | accuracy: 0.673 ± .003 -> 0.642 ± .003 | Hurt |
| increase encoder complexity from linear to convolutional at β=1 (ablation) | P-VAE · van Hateren | lifetime sparsity: 1.7× greater sparsity -> 2.4× greater sparsity | Helped |
| use ReLU-activated Gaussian latents (ablation) | Gaussian VAE · van Hateren | lifetime sparsity: s = 0.37 -> s = 0.69 | Helped |
Surprising findings
- Hard-forward surrogate gradients with integer latents in the forward pass underperformed the relaxed Poisson method.
- For low latent dimensionality (K=10) with logistic regression, P-VAE underperforms Gaussian and Laplace VAEs, so its advantage depends on dimensionality and classifier.
Limitations
- Real neuronal spike counts deviate from Poisson over longer windows and are super-Poisson in higher areas.
- Amortization gap between the convolutional P-VAE encoder and iterative LCA sparse coding remains.
- Results are on small datasets (natural image patches and MNIST), with hierarchical extensions left for future work.
Experimental setup
Datasets: van Hateren natural images CIFAR10 (16x16 patches) MNIST
Benchmarks: MNIST KNN classification (N=200, 1000, 5000 labels) MNIST shattering dimensionality van Hateren validation loss CIFAR_16x16 validation loss
Metrics: validation loss percent drop lifetime sparsity MSE proportion of active neurons KNN accuracy shattering dimensionality
Baselines: Gaussian VAE Laplace VAE Categorical VAE LCA ISTA Gaussian VAE with relu Gaussian VAE with exp
Builds on: variational autoencoder sparse coding predictive coding Gumbel-softmax / Categorical VAE straight-through estimator
Closest papers by meaning
- Trading Information between Latents in Hierarchical Variational Autoencoders (ICLR 2023 · similarity 0.89)
- Consistency Regularization for Variational Auto-Encoders (NeurIPS 2021 · similarity 0.89)
- Variational Autoencoders with Jointly Optimized Latent Dependency Structure (ICLR 2019 · similarity 0.89)
- Brain-like Variational Inference (NeurIPS 2025 · similarity 0.88)
- Multi-modal Gaussian Process Variational Autoencoders for Neural and Behavioral Data (ICLR 2024 · similarity 0.88)
- Beyond Vanilla Variational Autoencoders: Detecting Posterior Collapse in Conditional and Hierarchical Variational Autoencoders (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.