Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Scale-Invariant Empirical-Bayes Laplace Approximation for ReLU Networks
Shivam Pal, Piyush Rai
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a scale-invariant empirical-Bayes Laplace approximation for ReLU networks that removes dependence on the representative of the positive-homogeneity scale orbit.
Problem
Standard empirical-Bayes Laplace approximation for ReLU networks is not invariant to function-preserving rescaling of parameters, because positive homogeneity creates continuous scale orbits of equivalent parameters while the isotropic objective depends on the chosen representative through prior nor
Approach
Makes the post-hoc Bayesian evidence and uncertainty estimates invariant to function-preserving reparameterizations by accounting for the continuous scale orbits induced by ReLU positive homogeneity, rather than using the standard isotropic prior and objective.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply gamma-canonicalization before EB Laplace | diag-GGN EB-Laplace · Fashion-MNIST MLP orbit stress test | std of log lambda*: 0.356 -> 0 | Improved |
| Apply gamma-canonicalization before EB Laplace | diag-GGN EB-Laplace · Fashion-MNIST MLP orbit stress test | NLL: 0.3906 -> 0.3644 | Improved |
| Apply gamma-canonicalization before EB Laplace | diag-GGN EB-Laplace · Fashion-MNIST MLP orbit stress test | ECE: 0.0682 -> 0.0389 | Improved |
| Vary gamma from 0 to 1 in canonicalization (ablation) | gamma-canonicalized EB · Fashion-MNIST MLP orbit stress test | ECE: 0.0387 -> 0.0562 | Mixed |
| Apply gamma-canonicalization before KFAC-Laplace | KFAC-Laplace · CIFAR-10 ConvNet | NLL: 0.877 -> 0.734 | Improved |
| Apply gamma-canonicalization before KFAC-Laplace | KFAC-Laplace · CIFAR-10 ConvNet | ECE: 0.248 -> 0.130 | Improved |
| Apply gamma-canonicalization before KFAC-Laplace | KFAC-Laplace · CIFAR-10 ConvNet | accuracy: 0.774 -> 0.773 | No clear effect |
| Apply gamma-canonicalization before last-layer EB-Laplace on rescaled ResNet-18 | last-layer EB-Laplace · CIFAR-100 ResNet-18 | NLL: 1.1478 -> 0.7912 | Improved |
| Apply gamma-canonicalization to the original (well-scaled) ResNet representative | last-layer EB-Laplace · CIFAR-100 ResNet-18 | NLL: 0.7906 -> 0.7912 | No clear effect |
| Rescale the width-64 MLP by sigma=1.0 before standard EB | standard EB model selection · CIFAR-10 MLP width 64 vs 128 | ranking reversal rate: 0.0% -> 100.0% | Hurt |
Experimental setup
Baselines: standard empirical-Bayes Laplace approximation
Builds on: Laplace approximation empirical Bayes
Closest papers by meaning
- Reparameterization invariance in approximate Bayesian inference (NeurIPS 2024 · similarity 0.87)
- Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks (ICML 2020 · similarity 0.87)
- FSP-Laplace: Function-Space Priors for the Laplace Approximation in Bayesian Deep Learning (NeurIPS 2024 · similarity 0.86)
- Bayesian Treatment of the Spectrum of the Empirical Kernel in (Sub)Linear-Width Neural Networks (ICLR 2025 · similarity 0.86)
- Sampling-based inference for large linear models, with application to linearised Laplace (ICLR 2023 · similarity 0.85)
- Empirical Priors for Prediction in Sparse High-dimensional Linear Regression (JMLR 2020 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
