Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Rethinking Approximate Gaussian Inference in Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Replaces softmax with element-wise normCDF or sigmoid so that predictives from logit-space Gaussians (Laplace, HET, SNGP) have closed-form sample-free approximations, plus moment-matched Dirichlet outputs and BCE training.
Problem
Predictive probabilities from approximate Gaussian inference methods over logits require Monte Carlo sampling through softmax, which is costly (scales with number of classes) and noisy at inference.
Approach
Use element-wise normCDF or sigmoid followed by normalisation as the output activation; the Gaussian integrals then have closed-form (or probit-approximated) forms, giving sample-free predictives, moment-matched Dirichlet second-order distributions, and BCE training compatible with the framework.
Key results
- Closed-form predictives are on par with 1000-sample MC predictives on ImageNet while avoiding sampling overhead.
- On ImageNet, closed-form predictives clearly beat softmax on ECE across all methods.
- Moment-matched Dirichlet uncertainty gives best OOD AUROC (0.6388 for Sigmoid Laplace at ImageNet-C severity 1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace MC 1000 predictive with the closed-form predictive | Sigmoid Laplace · ImageNet | ECE: 0.0153 ± 0.0011 -> 0.0101 ± 0.0006 | Helped |
| Replace MC 1000 predictive with the closed-form predictive | NormCDF Laplace · ImageNet | ECE: 0.0118 ± 0.0007 -> 0.0114 ± 0.0007 | No clear effect |
| Use Dirichlet mutual information instead of MC expected entropy | Sigmoid Laplace · ImageNet-C severity 1 | OOD AUROC: 0.6321 ± 0.0001 -> 0.6388 ± 0.0005 | Helped |
| Use Dirichlet expected entropy instead of MC expected entropy | NormCDF HET · ImageNet-C severity 1 | OOD AUROC: 0.6281 ± 0.0020 -> 0.6353 ± 0.0021 | Helped |
| Use Dirichlet expected entropy instead of MC predictive entropy | NormCDF Laplace · ImageNet-C severity 1 | OOD AUROC: 0.6296 ± 0.0012 -> 0.6321 ± 0.0009 | Helped |
| Use closed-form predictive entropy instead of MC predictive entropy | NormCDF Laplace · ImageNet-C severity 1 | OOD AUROC: 0.6296 ± 0.0012 -> 0.6296 ± 0.0012 | No clear effect |
Surprising findings
- Softmax-based MC predictives are outperformed by closed-form normCDF/sigmoid predictives even at large sample counts, suggesting the activation change itself helps beyond computational savings.
Limitations
- Approximate predictives and Dirichlets do not encode correlations between classes.
- Theoretical bound relies on compactness assumptions on logit means and variances and on a probit approximation error term.
Experimental setup
Datasets: ImageNet-1k CIFAR-10 CIFAR-100
Benchmarks: ImageNet-1k CIFAR-10 CIFAR-100 ImageNet-C CIFAR-10C
Metrics: negative log likelihood expected calibration error correctness prediction log probability OOD detection AUROC KL divergence to MC predictive accuracy
Baselines: Softmax Monte Carlo predictive Laplace Bridge Mean Field approximation linear-time softmax approximation
Builds on: probit approximation Laplace approximation heteroscedastic classifiers SNGP Laplace bridge
Closest papers by meaning
- SoftCVI: Contrastive variational inference with self-generated soft labels (ICLR 2025 · similarity 0.84)
- Density-Softmax: Efficient Test-time Model for Uncertainty Estimation and Robustness under Distribution Shifts (ICML 2024 · similarity 0.84)
- The Minimax Rate of Second-Order Calibration (NeurIPS 2026 · similarity 0.84)
- A Study of Bayesian Neural Network Surrogates for Bayesian Optimization (ICLR 2024 · similarity 0.84)
- Adaptive Sampling for Efficient Softmax Approximation (NeurIPS 2024 · similarity 0.84)
- Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.