Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Microcanonical Langevin Ensembles: Advancing the Sampling of Bayesian Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Microcanonical Langevin Ensembles (MILE), which initializes MCLMC chains from deep-ensemble optimized solutions and adapts MCLMC tuning to sample Bayesian neural networks up to an order of magnitude faster than NUTS-based Bayesian Deep Ensembles.
Problem
Sampling-based inference for Bayesian neural networks is slow and unreliable because samplers struggle with multimodal, high-dimensional posteriors, poor initialization, and unpredictable NUTS runtimes.
Approach
Combine deep-ensemble optimization for chain initialization with the MCLMC sampler (no Metropolis adjustment, step size set via EEVPD), using a linear energy-variance scheduler, a step size initialized from the optimizer learning rate, and a fixed deterministic gradient budget per chain for predictable runtime.
Key results
- On UCI regression, MILE matches or beats BDE on LPPD and RMSE in most datasets while sampling time is roughly 7-9x lower than BDE (e.g., protein 19.5 min vs 152.85 min).
- Naive MCLMC yields 78-86% NaN chains on UCI regression datasets, while MILE shows 0% failures.
- On Fashion-MNIST CNNv2, MILE reaches 0.925 accuracy and -0.216 LPPD in about 1h 21min, versus Symmetric Split HMC at 0.813-0.820 accuracy and 7-19 hours.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace BDE's NUTS sampling with MILE | Bayesian neural networks · UCI Airfoil, Bikesharing, Concrete, Energy, Protein, and Yacht | LPPD: 0.558 -> 0.612; 0.625 -> 0.645; 0.301 -> 0.336; 2.072 -> 2.300; -0.760 -> -0.750; 2.674 -> 2.859 | Helped |
| Replace BDE's NUTS sampling with MILE | Bayesian neural networks · UCI Airfoil, Bikesharing, Concrete, Energy, Protein, and Yacht | RMSE: 0.214 -> 0.206; 0.242 -> 0.236; 0.273 -> 0.250; 0.045 -> 0.034; 0.703 -> 0.702; 0.083 -> 0.033 | Helped |
| Replace BDE's NUTS sampling with MILE | Bayesian neural networks · UCI Airfoil, Bikesharing, Concrete, Energy, Protein, and Yacht | Time (min): 2.25 -> 0.84; 48.29 -> 5.40; 1.56 -> 0.77; 1.11 -> 0.75; 152.85 -> 19.50; 0.58 -> 0.64 | Mixed |
| Remove MILE's proposed numerical-stability adjustments and use naive MCLMC (ablation) | MILE · Airfoil | NaN chains (%): 0% -> 86% | Hurt |
| Remove MILE's proposed numerical-stability adjustments and use naive MCLMC (ablation) | MILE · Concrete | NaN chains (%): 0% -> 80% | Hurt |
| Remove MILE's proposed numerical-stability adjustments and use naive MCLMC (ablation) | MILE · Energy | NaN chains (%): 0% -> 78% | Hurt |
| Remove MILE's proposed numerical-stability adjustments and use naive MCLMC (ablation) | MILE · Yacht | NaN chains (%): 0% -> 85% | Hurt |
| Extend the warmup phase (ablation) | MILE · Bikesharing | Hold-out performance | Helped |
| Use a linear energy-variance schedule instead of a fixed desired energy-variance level (ablation) | MILE · BNN sampling | Predictive performance | No clear effect |
| Use an exponentially decaying energy-variance schedule instead of the linear schedule (ablation) | MILE · BNN sampling | Predictive performance | Hurt |
Surprising findings
- Naive MCLMC fails numerically in the large majority of chains on BNN posteriors, so the modifications are essential rather than optional.
- Within-chain variance is largest in the first and last layers, suggesting multimodality is concentrated there rather than growing combinatorially with depth.
Limitations
- Omits Metropolis-Hastings adjustment, so the sampling distribution is biased, with convergence guarantees holding only under strict conditions.
- Prior choice is not studied in depth; larger models use tuned prior variances.
- Not designed for large-scale datasets; stochastic-gradient variants are left to future work.
Experimental setup
Datasets: Airfoil Bikesharing Concrete Energy Protein Yacht Ionosphere Income IMDB MNIST Fashion-MNIST UCI classification datasets (Sonar, WineWhite, WineRed, Australian, Heart, Glass, Covertype)
Benchmarks: UCI regression benchmark Ionosphere Income IMDB MNIST Fashion-MNIST UCI classification datasets (Sonar, WineWhite, WineRed, Australian, Heart, Glass, Covertype)
Metrics: LPPD RMSE accuracy wallclock time effective sample size chainwise split R-hat calibration error failure rate (NaN chains)
Baselines: Bayesian Deep Ensembles (NUTS) Deep Ensembles Path-Guided Particle-based Sampling Symmetric Split HMC naive MCLMC
Builds on: Microcanonical Langevin Monte Carlo Bayesian Deep Ensembles Deep Ensembles No-U-Turn Sampler
Closest papers by meaning
- Parameter Expanded Stochastic Gradient Markov Chain Monte Carlo (ICLR 2025 · similarity 0.89)
- Metropolis Adjusted Microcanonical Hamiltonian Monte Carlo (NeurIPS 2025 · similarity 0.88)
- A Study of Bayesian Neural Network Surrogates for Bayesian Optimization (ICLR 2024 · similarity 0.88)
- Reparameterized Importance Sampling for Robust Variational Bayesian Neural Networks (ICML 2024 · similarity 0.88)
- Sample as you Infer: Predictive Coding with Langevin Dynamics (ICML 2024 · similarity 0.88)
- Scalable Discrete Diffusion Samplers: Combinatorial Optimization and Statistical Physics (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.