Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
FSP-Laplace: Function-Space Priors for the Laplace Approximation in Bayesian Deep Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FSP-Laplace, a Laplace approximation for neural networks whose training objective regularizes the network in function space with a Gaussian process prior via an RKHS norm, scaled by matrix-free Lanczos linear algebra.
Problem
Isotropic Gaussian weight-space priors used in Laplace approximations are uninterpretable and pathological in deep networks, and existing function-space prior methods (VI) have infinite KL divergences, so informative priors are hard to specify for BNNs.
Approach
Recast training as finding the weak mode of the posterior measure under a GP prior restricted to the function class of the network, relax the hard constraint via a distance penalty, estimate the RKHS norm with context points resampled each step, linearize the network, and apply a matrix-free low-rank Laplace approximation.
Key results
- On Mauna Loa, FSP-Laplace reduces MSE to 8.90 versus 28.26 for Laplace and 38.38 for FVI.
- On MNIST with KMNIST context points, FSP-Laplace achieves the highest OOD detection accuracy (0.977) and log-likelihood -0.043 comparable to Laplace; on FashionMNIST it reaches 0.994 OOD accuracy versus 0.975 for FVI.
- On ocean currents, FSP-Laplace attains expected log-likelihood 0.2823 versus -0.5069 for GP and -126845 for Laplace.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the informative GP prior and periodic features instead of the standard Laplace setup | FSP-Laplace · Mauna Loa | mean squared error: 28.26 ± 6.87 -> 8.90 ± 1.15 | Helped |
| use the informative GP prior and periodic features instead of the standard Laplace setup | FSP-Laplace · Mauna Loa | expected log-likelihood: -16'139.85 ± 2'433.70 -> -6'015.39 ± 443.13 | Hurt |
| incorporate the Helmholtz kernel prior and neural-network parameterization | FSP-Laplace · Ocean current | expected log-likelihood: -126'845.4063 ± 14'470.5270 -> 0.2823 ± 0.0009 | Helped |
| incorporate the Helmholtz kernel prior and neural-network parameterization | FSP-Laplace · Ocean current | mean squared error: 0.0320 ± 0.0016 -> 0.0169 ± 0.0001; 0.0131 ± 0.0000 | Mixed |
| use KMNIST rather than RND context points | FSP-Laplace · MNIST | OOD accuracy: 0.895 ± 0.011 -> 0.977 ± 0.003 | Helped |
| use KMNIST rather than RND context points | FSP-Laplace · FashionMNIST | OOD accuracy: 0.797 ± 0.007 -> 0.994 ± 0.001 | Helped |
| use KMNIST rather than RND context points | FSP-Laplace · MNIST | log-likelihood: -0.037 ± 0.001 -> -0.043 ± 0.001 | Hurt |
| use KMNIST rather than RND context points | FSP-Laplace · FashionMNIST | accuracy: 0.909 ± 0.001 -> 0.909 ± 0.001 | No clear effect |
| use an interpretable GP function-space prior instead of an isotropic weight-space prior | FSP-Laplace · MNIST | accuracy: 0.989 ± 0.001 -> 0.989 ± 0.000 | No clear effect |
| use KMNIST context points | FSP-Laplace · FashionMNIST | ECE: 0.014 ± 0.002 -> 0.018 ± 0.002 | Hurt |
Surprising findings
- On Mauna Loa, the exact GP baseline has a better expected log-likelihood than all neural network methods, which under-estimate likelihood scale.
- Using RND context points, FSP-Laplace's OOD detection drops to roughly Laplace level, showing sensitivity to context point distribution.
Limitations
- Performance depends on having prior knowledge expressible as a GP kernel; less advantageous for high-dimensional inputs like images where good function-space priors are unclear.
- Conjecture that the relaxed weak modes are weak modes of the true posterior is left unproven.
- Requires a sufficient number of context points; the variance truncation heuristic is needed to avoid exploding predictive variance.
Experimental setup
Datasets: Mauna Loa CO2 GulfDrifters MNIST FashionMNIST KMNIST UCI regression scikit-learn two moons song dataset
Benchmarks: Mauna Loa CO2 GulfDrifters ocean currents MNIST FashionMNIST UCI regression Bayesian optimization benchmarks OOD detection (MNIST vs FashionMNIST)
Metrics: expected log-likelihood mean squared error accuracy expected calibration error OOD detection accuracy mean rank
Baselines: MAP Laplace FVI GP sparse GP
Builds on: linearized Laplace approximation function-space variational inference (FVI) weak mode of measures Kronecker-factored Laplace (K-FAC)
Closest papers by meaning
- Function-space Parameterization of Neural Networks for Sequential Learning (ICLR 2024 · similarity 0.88)
- Variational Linearized Laplace Approximation for Bayesian Deep Learning (ICML 2024 · similarity 0.86)
- Laplace Sample Information: Data Informativeness Through a Bayesian Lens (ICLR 2025 · similarity 0.85)
- Reparameterization invariance in approximate Bayesian inference (NeurIPS 2024 · similarity 0.85)
- Debiasing Mini-Batch Quadratics for Applications in Deep Learning (ICLR 2025 · similarity 0.85)
- Bayesian Optimization on Function Spaces via Sparse RKHS Manifolds (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.