Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IBDR, a Bayesian particle-based fine-tuning framework with a joint particle divergence (determinant/volume) loss and Wasserstein distributionally robust (SAM-like) optimization, evaluated on VTAB-1K with ViT-B/16 and commonsense reasoning with LLaMA2-7B.
Problem
Independent posterior particle sampling in Bayesian fine-tuning fails to capture particle interactions and can collapse to a single mode, limiting ensemble diversity and quality for foundation-model fine-tuning.
Approach
Model interactions between particles during training via a joint approximate posterior over K particles, add a determinant-based divergence loss on non-maximal predictions, and combine with Wasserstein distributionally robust optimization (a SAM-like inner ascent step) and a generalized PAC-Bayes bound; particles are sampled independently at inference.
Key results
- On VTAB-1K with ViT-B/16, IBDR reaches 73.6% average top-1 accuracy, versus 70.9% for SVGD and 68.2% for SA-BNN.
- On six commonsense reasoning datasets with LLaMA2-7B, IBDR averages 79.66% accuracy, above BLoB (78.75%).
- IBDR's average ECE on commonsense reasoning is 12.27, slightly worse than LAP (11.85) and BLoB (12.12).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the divergence loss for particle interaction (α=0.02 rather than α=0) (ablation) | IBDR · DTD | Accuracy: 67.24 -> 71.70 | Helped |
| add the divergence loss for particle interaction (α=0.02 rather than α=0) (ablation) | IBDR · DMLab | Accuracy: 51.92 -> 53.61 | Helped |
| add the divergence loss for particle interaction (α=0.02 rather than α=0) (ablation) | IBDR · SVHN | Accuracy: 87.03 -> 91.32 | Helped |
| increase particle count from 1 to 4 (ablation) | IBDR · Camelyon | Classification accuracy: 82.4 -> 85.1 | Helped |
| increase particle count from 4 to 8 (ablation) | IBDR · Camelyon | Classification accuracy: 85.1 -> 85.8 | Helped |
| increase particle count from 4 to 8 (ablation) | IBDR · EuroSAT | Classification accuracy: 95.0 -> 95.5 | Helped |
| increase particle count from 4 to 8 (ablation) | IBDR · Resics45 | Classification accuracy: 87.3 -> 87.4 | Helped |
| increase particle count from 4 to 8 (ablation) | IBDR · Retinopathy | Classification accuracy: 76.5 -> 77.0 | Helped |
| increase diversity weight α from 0.02 to 0.08 (ablation) | IBDR · DTD | Accuracy: 71.70 -> 71.73 | Helped |
| increase diversity weight α from 0.02 to 0.08 (ablation) | IBDR · DMLab | Accuracy: 53.61 -> 52.73 | Hurt |
| increase diversity weight α from 0.02 to 0.08 (ablation) | IBDR · SVHN | Accuracy: 91.32 -> 91.26 | Hurt |
| increase particle count from 4 to 8 (ablation) | IBDR · VTAB-1K specialized datasets | Runtime per epoch: $158_{\pm 4.3}$ -> $220_{\pm 6.2}$ | Hurt |
Surprising findings
- On commonsense reasoning, IBDR has worse ECE than several baselines (e.g., 24.17 on WG-S vs. 4.15 for LAP), showing calibration trade-offs.
Limitations
- Runtime and memory grow linearly with the number of particles.
- Larger divergence weight alpha can destabilize training; hyperparameter sensitivity to rho and alpha.
- Calibration is not uniformly better than baselines on commonsense reasoning.
Experimental setup
Datasets: VTAB-1K ImageNet-21K ARC-Challenge ARC-Easy Winogrande-Small Winogrande-Medium OpenBookQA BoolQ
Benchmarks: VTAB-1K ARC-C ARC-E WG-S WG-M OBQA BoolQ
Metrics: top-1 accuracy expected calibration error OOD detection percentage
Baselines: FFT LoRA SAM SA-BNN SGLD DeepEns BayesTune SVGD MLE MAP MCD BBB
Builds on: Sharpness-Aware Minimization SA-BNN Stein Variational Gradient Descent LoRA BLoB
Closest papers by meaning
- To discretize continually: Mean shift interacting particle systems for Bayesian inference (NeurIPS 2026 · similarity 0.86)
- Structure-informed Risk Minimization for Robust Ensemble Learning (ICML 2025 · similarity 0.84)
- GAD-PVI: A General Accelerated Dynamic-Weight Particle-Based Variational Inference Framework (AAAI 2024 · similarity 0.83)
- Measuring Stochastic Data Complexity with Boltzmann Influence Functions (ICML 2024 · similarity 0.83)
- A Mean-Field Framework for Inference-Time Distributional Control of Diffusion Models (NeurIPS 2026 · similarity 0.83)
- Particle Guidance: non-I.I.D. Diverse Sampling with Diffusion Models (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.