Research map / Probabilistic and scientific modeling / Variational and generative sampling
Hamiltonian Monte Carlo on ReLU Neural Networks is Inefficient
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that leapfrog HMC on ReLU-based Bayesian neural networks incurs Omega(epsilon) local energy error at activation-boundary crossings (vs O(epsilon^3) for smooth targets), derives d^{-1/2} step-size scaling and ~0.45 optimal acceptance, and validates with synthetic and UTKFace experiments.
Problem
HMC with leapfrog integration is assumed to be efficient for Bayesian neural networks, but ReLU's non-differentiability creates kinks in the potential energy, which may break the classical error analysis and cause low acceptance rates.
Approach
Shows leapfrog HMC remains correct for ReLU networks under deterministic derivative definitions, but each crossing of an activation boundary yields a local energy error of order Omega(epsilon) rather than O(epsilon^3), so acceptance decays; derives revised tuning guidelines (step size d^{-1/2}, optimal acceptance about 0.45) for piecewise-affine i.i.d. targets.
Key results
- On the synthetic data with one 50-node hidden layer, ReLU and leaky ReLU acceptance rates collapse at step size 0.0020-0.0025 (near 0) while sigmoid networks keep ~0.85-0.92 acceptance.
- Empirically optimal acceptance rates were 0.755 (sigmoid), 0.6525 (ReLU), 0.724 (leaky ReLU) on synthetic data; on UTKFace the optimal rates were 0.813, 0.797, 0.627.
- On UTKFace, sigmoid networks reached mean acceptance 0.5451 vs 0.2239 for ReLU and 0.1757 for leaky ReLU, with best-acceptance test MSE nearly identical (~0.053-0.054).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use sigmoid rather than ReLU activation | Bayesian neural network with HMC · UTKFace test | average acceptance rate: 0.2239 $\pm$ 0.0617 -> 0.5451 $\pm$ 0.0687 | Helped |
| Use sigmoid rather than ReLU activation | Bayesian neural network with HMC · UTKFace test | average MSE (overall): 0.1153 $\pm$ 0.0057 -> 0.0552 $\pm$ 0.0003 | Helped |
| Tune each activation to its best empirical acceptance rate | UTKFace Bayesian neural networks with HMC · UTKFace test | average MSE at best acceptance rate: 0.0539 $\pm$ 0.0001; 0.0533 $\pm$ 0.0004; 0.0538 $\pm$ 0.0002 | No clear effect |
| Use sigmoid rather than ReLU activation | Bayesian neural network with HMC · Synthetic dataset (L=200, ε=0.0010) | acceptance rate: 0.652 $\pm$ 0.019 -> 0.981 $\pm$ 0.002 | Helped |
| Use sigmoid rather than leaky ReLU activation | Bayesian neural network with HMC · Synthetic dataset (L=200, ε=0.0010) | acceptance rate: 0.653 $\pm$ 0.014 -> 0.981 $\pm$ 0.002 | Helped |
Surprising findings
- Once tuned to the best acceptance rate, ReLU, leaky ReLU and sigmoid networks achieve nearly identical test MSE on UTKFace despite the large acceptance gap.
- Wider shallow networks showed lower HMC acceptance than deeper networks with the same parameter count.
Limitations
- Theoretical optimal-scaling results (d^{-1/2}, 0.45) hold only for a proxy model of i.i.d. piecewise-affine components, not general ReLU networks.
- Number of non-differentiable crossings for general ReLU networks in parameter space is not characterized.
- Experiments are small-scale (100-200 training points, single hidden layer or small MLPs) on a single CPU.
Experimental setup
Datasets: UTKFace (male Asian subset) synthetic regression data (x~U(0,4), y~N(cos 2x, 0.1^2))
Benchmarks: UTKFace subset (age regression) synthetic cos(2x) regression
Metrics: acceptance rate efficiency (epsilon * acceptance) test MSE
Baselines: sigmoid networks leaky ReLU networks classical smooth-HMC tuning (0.651 acceptance, d^{-1/4})
Builds on: Hamiltonian Monte Carlo Roberts-Rosenthal optimal scaling Beskos et al. optimal tuning of HMC
Closest papers by meaning
- Generalizing Hamiltonian Monte Carlo with Neural Networks (ICLR 2018 · similarity 0.88)
- Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks (ICML 2020 · similarity 0.87)
- Entropy-based adaptive Hamiltonian Monte Carlo (NeurIPS 2021 · similarity 0.86)
- A Study of Bayesian Neural Network Surrogates for Bayesian Optimization (ICLR 2024 · similarity 0.85)
- Microcanonical Langevin Ensembles: Advancing the Sampling of Bayesian Neural Networks (ICLR 2025 · similarity 0.85)
- Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networks (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.