Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Improving Generalization with Flat Hilbert Bayesian Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Flat Hilbert Bayesian Inference (FHBI), a SAM-style functional sharpness-aware particle sampler in RKHS that targets the population posterior, evaluated with LoRA on ViT-B/16 over VTAB-1K.
Problem
Bayesian inference over neural network parameters typically approximates the empirical posterior, which overfits; the goal is to approximate the population posterior to improve generalization.
Approach
Extends sharpness-aware generalization bounds from Euclidean space to RKHS functional space and derives a two-step (adversarial functional perturbation + functional descent) SVGD-style particle update, which generalizes SVGD (rho=0) and SAM (one particle).
Key results
- FHBI achieves 73.7% average top-1 accuracy on VTAB-1K across 19 tasks, versus 72.2% for BayesTune, 70.5% for SAM and 70.9% for SVGD (ViT-B/16 with LoRA, 4 particles).
- FHBI reports average ECE of 0.12 on VTAB-1K, the lowest among the compared methods (SVGD 0.18, SAM 0.16, BayesTune 0.23).
- Increasing particles from 1 (SAM) to 4 raised CIFAR-100 accuracy from 72.7 to 74.8 in the particle ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FHBI (functional sharpness-aware particle update in RKHS, 4 particles) instead of SVGD | LoRA Bayesian fine-tuning on ViT-B/16 · VTAB-1K | Top-1 accuracy AVG: 73.7 vs 70.9 | Helped |
| Use FHBI instead of SAM (single model) | LoRA fine-tuning on ViT-B/16 · VTAB-1K | Top-1 accuracy AVG: 73.7 vs 70.5 (3.2 points) | Helped |
| Use multiple particles (FHBI 4p) instead of single particle SAM (ablation) | CIFAR100 natural subset · VTAB-1K Natural datasets | Top-1 accuracy (CIFAR100): 74.8 vs 72.7 | Helped |
| Increase particles from 4 to 10 (ablation) | FHBI · VTAB-1K Natural datasets | Top-1 accuracy: small gains on most datasets (e.g., CIFAR100 74.8 to 75.0), Flower102 drops 99.4 to 99.1 | Mixed |
| Use polynomial kernel (degree 10) instead of RBF kernel (ablation) | FHBI · Specialized datasets (4) | Top-1 accuracy AVG: 86.5 vs 86.8 | Hurt |
| Use FHBI instead of SVGD (sharpness-aware update lowers particle sharpness) (ablation) | Particle sharpness on KITTI · KITTI | particle sharpness evolution: lower and more stable sharpness than SVGD | Helped |
| Use FHBI instead of SVGD | Expected calibration error · VTAB-1K | ECE AVG: 0.12 vs 0.18 (SVGD) | Helped |
Surprising findings
- Reported ECE trade-off: the authors note accuracy and ECE typically trade off, yet FHBI improves both.
- BayesTune reports an ECE of 0.85 on Pets, an outlier value in the baseline table.
Limitations
- Stores multiple model particles, a memory bottleneck for larger models.
- Hyperparameters (kernel width, learning rate, ascent step) tuned by grid search on validation sets per dataset.
Experimental setup
Datasets: VTAB-1K
Benchmarks: VTAB-1K
Metrics: Top-1 accuracy Expected Calibration Error
Baselines: FFT AdamW SAM DeepEns BayesTune SGLD SADA-JEM SA-BNN SVGD
Builds on: Stein Variational Gradient Descent Sharpness-Aware Minimization LoRA SA-BNN SADA-JEM
Closest papers by meaning
- Kernel Bayesian Inference with Posterior Regularization (NeurIPS 2016 · similarity 0.86)
- Bayesian Online Natural Gradient (BONG) (NeurIPS 2024 · similarity 0.85)
- Kernel Semi-Implicit Variational Inference (ICML 2024 · similarity 0.85)
- Leveraging Self-Consistency for Data-Efficient Amortized Bayesian Inference (ICML 2024 · similarity 0.84)
- Scalable Bayesian Learning with posteriors (ICLR 2025 · similarity 0.84)
- VIKING: Deep variational inference with stochastic projections (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.