Research map / Large language models / Instruction and fine-tuning
BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BLoB, which jointly learns mean and covariance of a low-rank variational posterior over LoRA's A matrix via backpropagation during fine-tuning, improving accuracy and calibration of LLMs on in- and out-of-distribution commonsense/science QA.
Problem
LLMs fine-tuned with LoRA on small domain-specific data are overconfident; existing Bayesian uncertainty methods for LoRA either ensemble or apply post-hoc Laplace approximations after MAP training, leading to suboptimal posterior estimates.
Approach
Bayesianize only the LoRA A matrix with a Gaussian variational posterior, show the induced full-weight posterior is a low-rank Gaussian, derive a closed-form KL computable in the low-rank space, and jointly optimize mean and variance throughout fine-tuning using a squared-std parameterization and Flipout.
Key results
- BLoB (N=10) reduces ECE on WG-S from 29.83 (MLE) to 9.35 and on ARC-E to 3.64 while keeping accuracy comparable to MLE.
- BLoB (N=0) attains the best accuracy on WG-S (70.89), ARC-C (70.83), ARC-E (86.68), and OBQA (82.73) among compared methods.
- On out-of-distribution shifts (ARC-C, ARC-E, MMLU chemistry/physics), BLoB achieves the best or second-best NLL in smaller shifts and remains competitive with ENS and LAP under larger shifts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BLoB (N=10) instead of the strongest baseline | Llama2-7B LoRA fine-tuning · ARC-C | ECE: 16.25 -> 9.59 | Helped |
| Use BLoB (N=10) instead of the strongest baseline | Llama2-7B LoRA fine-tuning · ARC-E | ECE: 12.28 -> 3.64 | Helped |
| Use BLoB (N=10) instead of the strongest baseline | Llama2-7B LoRA fine-tuning · WG-M | ECE: 7.40 -> 3.01 | Helped |
| Use BLoB (N=10) instead of the strongest baseline | Llama2-7B LoRA fine-tuning · OBQA | ECE: 11.38 -> 3.77 | Helped |
| Use BLoB (N=10) instead of the strongest baseline | Llama2-7B LoRA fine-tuning · BoolQ | ECE: 1.30 -> 1.41 | Hurt |
| Use BLoB (N=10) instead of the strongest baseline | Llama2-7B LoRA fine-tuning · WG-S | ECE: 4.15 -> 9.35 | Hurt |
| Use BLoB (N=10) instead of the strongest baseline | Llama2-7B LoRA fine-tuning · ARC-C | NLL: 1.03 -> 0.78 | Helped |
| Use BLoB (N=10) instead of the strongest baseline | Llama2-7B LoRA fine-tuning · ARC-E | NLL: 0.82 -> 0.40 | Helped |
| Use BLoB (N=10) instead of the strongest baseline | Llama2-7B LoRA fine-tuning · WG-S | NLL: 0.60 -> 0.63 | Hurt |
| Use BLoB (N=10) instead of the strongest baseline | Llama2-7B LoRA fine-tuning · BoolQ | NLL: 0.31 -> 0.31 | No clear effect |
| Use BLoB (N=10) instead of the strongest baseline | Llama2-7B LoRA fine-tuning · ARC-C | ACC: 69.10 -> 68.81 | Hurt |
| Use BLoB (N=10) instead of the strongest baseline | Llama2-7B LoRA fine-tuning · ARC-E | ACC: 86.55 -> 85.56 | Hurt |
Surprising findings
- BBB, a joint mean-covariance method, yields very low accuracy on WG-S (56.54), showing naive Bayesian training can hurt accuracy.
- Laplace-LoRA shows training failures on ARC-C and achieves very low ECE on WG-S but high ECE on ARC-E (33.29), indicating inconsistent post-training calibration.
Limitations
- Restricted to fine-tuning; not applicable to training-free uncertainty estimation.
- Requires multiple sampling passes at inference, which challenges efficient deployment.
- Evaluated only on classification-style tasks; generation tasks not studied.
Experimental setup
Datasets: WinoGrande-S WinoGrande-M ARC-Challenge ARC-Easy OpenBookQA BoolQ MMLU chemistry MMLU physics
Benchmarks: WinoGrande-S WinoGrande-M ARC-Challenge ARC-Easy OpenBookQA BoolQ MMLU chemistry MMLU physics
Metrics: accuracy expected calibration error negative log-likelihood
Baselines: MLE MAP Monte-Carlo Dropout Deep Ensemble Bayes By Backprop Laplace-LoRA
Builds on: LoRA Bayes By Backprop Flipout Laplace-LoRA
Closest papers by meaning
- Training-Free Bayesianization for Low-Rank Adapters of Large Language Models (NeurIPS 2025 · similarity 0.94)
- C-LoRA: Contextual Low-Rank Adaptation for Uncertainty Estimation in Large Language Models (NeurIPS 2025 · similarity 0.91)
- Bayesian Low-rank Adaptation for Large Language Models (ICLR 2024 · similarity 0.90)
- Large Language Models to Enhance Bayesian Optimization (ICLR 2024 · similarity 0.89)
- QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models (ICLR 2024 · similarity 0.89)
- Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.