Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Connecting the Dots: Is Mode-Connectedness the Key to Feasible Sample-Based Inference in Bayesian Neural Networks?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies sample-based inference (NUTS/HMC) for Bayesian neural networks, links posterior multimodality from overparameterization and activation choice to sampler failure, and proposes DEI-MCMC, which initializes MCMC chains from deep ensemble solutions.
Problem
Sample-based inference for Bayesian neural networks is hard because of the large, symmetric, multimodal parameter-space posterior, causing samplers to get stuck (dying samplers, especially with unbounded activations like ReLU) and making convergence diagnostics unreliable.
Approach
Connects posterior symmetries from overparameterization to sampling difficulty, finds deeper layers are better connected, proposes layer-wise and chain-wise convergence diagnostics, and initializes multiple NUTS chains from deep ensemble members (DEI-MCMC) to avoid dying samplers.
Key results
- Across tanh-activated BNNs, chains beating a linear model yield an ensemble that outperforms random forest, DNN and deep ensemble baselines in RMSE on several datasets (e.g., yacht RMSE 0.022 vs 0.072 RF and 0.103 DE).
- With ReLU activation, DEI-MCMC with 10 samples improves LPPD over deep ensembles on most datasets (e.g., airfoil 0.50 vs 0.19 DE) and avoids dying samplers, with 100x shorter warmup (100 steps).
- HMC with fixed hyperparameters yields better-than-LM chains in only about 1% of experiments, while NUTS performs much better, e.g., 58% of tanh chains on concrete.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Initialize ReLU BNN sampling from deep-ensemble solutions instead of cold starts | DEI-MCMC · airfoil, bikesharing, concrete, energy, protein, yacht | LPPD: 0.19±0.05 -> 0.58±0.03 (airfoil); 0.42±0.00 -> 0.63±0.02 (bikesharing); -0.08±0.20 -> 0.23±0.12 (concrete); 1.93±0.05 -> | Helped |
| Increase DEI-MCMC sampling from 10 to 1000 samples (ablation) | DEI-MCMC · airfoil, bikesharing, concrete, energy, protein, yacht | LPPD: 0.50±0.01 -> 0.58±0.03 (airfoil); 0.60±0.02 -> 0.63±0.02 (bikesharing); 0.05±0.27 -> 0.23±0.12 (concrete); 1.91±0.28 -> | Helped |
| Increase DEI-MCMC sampling from 10 to 1000 samples (ablation) | DEI-MCMC · airfoil, bikesharing, concrete, energy, protein, yacht | RMSE: 0.21±0.01 -> 0.21±0.01 (airfoil); 0.26±0.01 -> 0.25±0.01 (bikesharing); 0.40±0.19 -> 0.31±0.05 (concrete); 0.04±0.00 -> | Mixed |
| Use NUTS instead of fixed-hyperparameter HMC | BNN sampling · concrete and energy | proportion of chains beating LM: 0.00 -> 0.58 (concrete, tanh); 0.00 -> 0.67 (energy, tanh) | Helped |
| Use tanh instead of ReLU activation | BNN sampling with NUTS · concrete and energy | proportion of chains beating LM: 0.14 -> 0.58 (concrete); 0.31 -> 0.67 (energy) | Helped |
| Increase the number of chains and posterior samples | NUTS BNN · airfoil and other reported data sets | predictive coverage calibration | Helped |
| Extend warmup from 10,000 to up to 100,000 steps (ablation) | NUTS BNN · reported experiments | predictive performance | No clear effect |
Surprising findings
- Posterior modes become more connected in deeper hidden layers, while the first and last layers show disconnected modes.
- Smoothed ReLU variants (SiLU, leaky ReLU) offer little mitigation of the dying sampler problem; only truncated ReLU avoids it.
- Effective sample size is small for BNN parameters and classical split-R-hat aggregated over all weights falsely suggests convergence.
Limitations
- Only full-batch sampling routines were tested; no SG-MCMC or larger datasets.
- Out-of-distribution detection and other downstream uncertainty tasks not evaluated.
- Parameter-space ESS is small and judged not a useful convergence measure; diagnostics are heuristic.
Experimental setup
Datasets: UCI airfoil UCI concrete UCI energy UCI yacht UCI bikesharing UCI protein
Benchmarks: UCI airfoil UCI concrete UCI energy UCI yacht UCI bikesharing UCI protein
Metrics: RMSE log posterior predictive density (LPPD) credibility interval coverage R-hat ESS
Baselines: linear model random forest deep neural network deep ensemble random sample from BNN chains HMC with fixed hyperparameters
Builds on: No-U-Turn Sampler deep ensembles rank-normalized split-R-hat Laplace approximation
Closest papers by meaning
- Microcanonical Langevin Ensembles: Advancing the Sampling of Bayesian Neural Networks (ICLR 2025 · similarity 0.87)
- Reparameterized Importance Sampling for Robust Variational Bayesian Neural Networks (ICML 2024 · similarity 0.86)
- A Study of Bayesian Neural Network Surrogates for Bayesian Optimization (ICLR 2024 · similarity 0.86)
- Effortless, Simulation-Efficient Bayesian Inference using Tabular Foundation Models (NeurIPS 2025 · similarity 0.85)
- Training Bayesian Neural Networks with Sparse Subspace Variational Inference (ICLR 2024 · similarity 0.85)
- A Symmetry-Aware Exploration of Bayesian Neural Network Posteriors (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.