Research map / Probabilistic and scientific modeling / Variational and generative sampling
Entropy-MCMC: Sampling from Flat Basins with Ease
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Entropy-MCMC, which couples the model parameter with an auxiliary guiding variable whose stationary distribution is a local-entropy-smoothed posterior, so SGLD-style sampling is biased toward flat posterior basins at near-SGLD cost.
Problem
Bayesian deep learning samplers with limited budgets get trapped in sharp, poorly generalizing posterior modes; explicitly favoring flat basins usually requires expensive nested loops, doubled gradients, or min-max optimization.
Approach
Introduce an auxiliary guiding variable theta_a coupled to theta via a joint density proportional to exp(-f(theta) - ||theta - theta_a||^2/(2 eta)); its marginal for theta_a is the local-entropy smoothed posterior, and jointly running SGLD on (theta, theta_a) pulls theta toward flat basins without nested chains.
Key results
- On CIFAR100 with ResNet18, EMCMC reaches 79.16% accuracy and 0.840 NLL, versus 78.79% / 0.854 for SGLD and 76.49% for SGD.
- On ImageNet with ResNet50, EMCMC achieves 77.096% top-1 and 0.895 NLL, improving over SGLD (76.676%, 0.921).
- On CIFAR100-to-SVHN OOD detection, EMCMC reaches 81.14% AUROC versus 72.51% for SGLD and 79.15% for Entropy-SGD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EMCMC instead of bSAM | ResNet18 · CIFAR10 | accuracy (%): 95.53±0.09 -> 95.69±0.06 | Helped |
| Use EMCMC instead of SGLD | ResNet18 · CIFAR100 | accuracy (%): 78.79±0.35 -> 79.16±0.07 | Helped |
| Use EMCMC instead of SGLD | ResNet18 · Corrupted CIFAR | accuracy (%): 88.61 -> 88.87; 82.46 -> 83.27; 76.49 -> 77.44; 69.19 -> 70.31; 56.98 -> 58.17 | Helped |
| Use EMCMC instead of SGLD | ResNet50 · ImageNet | Top-1 accuracy (%): 76.676 -> 77.096 | Helped |
| Use EMCMC instead of Entropy-SGD | ResNet18 · CIFAR10-SVHN | AUROC (%): 98.71 -> 98.15 | Hurt |
| Use EMCMC instead of Entropy-SGD | ResNet18 · CIFAR100-SVHN | AUROC (%): 79.15 -> 81.14 | Helped |
| Use EMCMC instead of SAM | ResNet18 · CIFAR10 calibration | AUROC (%): 94.91 -> 93.76 | Hurt |
| Use EMCMC instead of Entropy-SGLD | ResNet18 · CIFAR100 calibration | ECE (%): 1.66 -> 3.59 | Hurt |
| Disable data augmentation (ablation) | EMCMC · CIFAR10 | accuracy (%): 89.87 -> 95.79 | Hurt |
| Reduce temperature from $10^0$ to $10^{-4}$ (ablation) | EMCMC · CIFAR10 | test accuracy (%): 81.93 -> 95.06 | Helped |
| Use cyclical rather than step step-size schedule (ablation) | EMCMC · CIFAR100 | testing accuracy (%): 89.59 -> 89.93 | Helped |
| Use joint rather than Gibbs-like updating (ablation) | EMCMC · classification experiments | performance | Helped |
Surprising findings
- Entropy-SGLD underperforms plain SGLD on CIFAR10 classification (94.46% vs 95.47%) and on CIFAR10-SVHN OOD detection (90.07 vs 97.66 AUROC).
- On CIFAR10-SVHN OOD detection, plain SGD and SGLD outperform EMCMC in AUROC (98.30 and 97.66 vs 98.15 for EMCMC).
Limitations
- Convergence guarantees hold only for strongly log-concave, smooth targets; non-log-concave case left for future work.
- Hyperparameter eta must be tuned (e.g., 1e-3 for CIFAR10, 1e-2 for CIFAR100), and the paper notes eta must satisfy m <= 1/eta <= M for the theory.
Experimental setup
Datasets: CIFAR10 CIFAR100 corrupted CIFAR ImageNet MNIST SVHN
Benchmarks: CIFAR10 CIFAR100 corrupted CIFAR ImageNet CIFAR-SVHN OOD
Metrics: accuracy NLL top-5 accuracy AUROC AUPR ECE Hessian eigenspectrum training NLL
Baselines: SGD Entropy-SGD SAM bSAM SGLD Entropy-SGLD
Builds on: SGLD Entropy-SGD local entropy
Closest papers by meaning
- Stable Coresets via Posterior Sampling: Aligning Induced and Full Loss Landscapes (NeurIPS 2025 · similarity 0.86)
- Parameter Expanded Stochastic Gradient Markov Chain Monte Carlo (ICLR 2025 · similarity 0.85)
- Scalable Discrete Diffusion Samplers: Combinatorial Optimization and Statistical Physics (ICLR 2025 · similarity 0.85)
- Provable Posterior Sampling with Denoising Oracles via Tilted Transport (NeurIPS 2024 · similarity 0.85)
- Divide-and-Conquer Posterior Sampling for Denoising Diffusion priors (NeurIPS 2024 · similarity 0.84)
- Microcanonical Langevin Ensembles: Advancing the Sampling of Bayesian Neural Networks (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.