Research map / Large language models / Instruction and fine-tuning
Bayesian Low-rank Adaptation for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Applies a post-hoc Kronecker-factored Laplace approximation to LoRA adapter weights of fine-tuned LLMs (Laplace-LoRA) to improve calibration without changing standard fine-tuning.
Problem
Fine-tuned LLMs are often overconfident, especially when fine-tuned on small datasets, and existing Bayesian methods are costly at LLM scale.
Approach
Apply a post-hoc Laplace approximation only over the low-rank LoRA adapter parameters (treated as linear layers with KFAC structure, using low-rank Kronecker factors), leaving the fine-tuning pipeline unchanged, and use linearized predictive sampling with the marginal likelihood to tune the prior precision.
Key results
- On LLaMA2-7B with 5000 steps (training-only setting), Laplace-LoRA (LA) reduced ECE on WG-S from 31.2 (MAP) to 2.1 and NLL from 3.15 to 0.60 while keeping accuracy similar.
- Under distribution shift (OBQA-trained LLaMA2-7B), LA reduced ECE on ARC-C from 22.2 (MAP) to 8.8 and on CS from 34.2 to 15.8.
- Laplace-LoRA adds only about 1-5% memory overhead and roughly 10% time overhead in the reported runs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply full Laplace approximation to all LoRA weights rather than standard MAP fine-tuning | LLaMA2-7B · Winogrande-small, ARC-Challenge, ARC-Easy, Winogrande-medium, OpenbookQA, BoolQ | NLL: 3.15_{0.10} -> 0.60_{0.01}; 3.28_{0.29} -> 0.88_{0.03}; 1.26_{0.13} -> 0.49_{0.06}; 1.51_{0.05} -> 0.63_{0.02}; 0.99_{0. | Helped |
| Apply full Laplace approximation to all LoRA weights rather than standard MAP fine-tuning | LLaMA2-7B · Winogrande-small, ARC-Challenge, ARC-Easy, Winogrande-medium, OpenbookQA, BoolQ | ECE: 31.2_{0.3} -> 2.1_{0.3}; 31.0_{0.5} -> 7.4_{0.7}; 13.4_{1.3} -> 5.4_{0.2}; 23.0_{0.1} -> 7.4_{0.4}; 16.1_{0.6} -> 6.4_{0 | Helped |
| Apply full Laplace approximation to all LoRA weights rather than MAP | LLaMA2-7B · Winogrande-small, ARC-Challenge, ARC-Easy, Winogrande-medium, OpenbookQA, BoolQ | accuracy: 67.4_{0.3} -> 67.3_{0.2}; 66.3_{0.6} -> 65.3_{0.2}; 84.7_{1.5} -> 85.1_{1.5}; 73.4_{0.4} -> 73.4_{0.3}; 78.7_{0.4} -> 78 | No clear effect |
| Use last-layer Laplace approximation on output-layer LoRA weights instead of all LoRA weights (ablation) | Laplace-LoRA · Common-sense reasoning benchmarks | ECE | Mixed |
| Use diagonal Fisher instead of KFAC for the Laplace Hessian approximation (ablation) | Laplace-LoRA · Common-sense reasoning and text classification tasks | ECE and NLL | Mixed |
| Use generalized probit, Laplace bridge, or diagonal-covariance Monte Carlo instead of full-covariance Monte Carlo (ablation) | Laplace-LoRA predictive posterior · Six common-sense reasoning tasks | accuracy and NLL | Mixed |
Surprising findings
- Checkpoint ensembles often performed much worse than MAP on RoBERTa models, in contrast to their behavior on LLaMA2-7B.
- Diagonal Fisher approximations did not give consistent ECE/NLL improvements, unlike KFAC which models weight correlations.
Limitations
- Gains are reported mostly on small common-sense and classification datasets; large-scale settings untested.
- Predictive approximations (probit, Laplace bridge) underperform Monte Carlo sampling, so the method relies on sampling at inference.
- Last-layer Laplace gives only modest gains on several tasks.
Experimental setup
Datasets: Winogrande ARC-Challenge ARC-Easy OpenBookQA BoolQ MMLU GLUE SuperGLUE
Benchmarks: Winogrande ARC-Challenge ARC-Easy OpenBookQA BoolQ MMLU subsets GLUE SuperGLUE
Metrics: accuracy expected calibration error negative log-likelihood
Baselines: MAP Monte-Carlo dropout checkpoint ensemble deep ensemble temperature scaling last-layer Laplace
Builds on: LoRA Laplace approximation KFAC linearized Laplace
Closest papers by meaning
- C-LoRA: Contextual Low-Rank Adaptation for Uncertainty Estimation in Large Language Models (NeurIPS 2025 · similarity 0.93)
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.92)
- Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning (NeurIPS 2025 · similarity 0.91)
- Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation (NeurIPS 2026 · similarity 0.91)
- QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models (ICLR 2024 · similarity 0.91)
- Loquetier: A Virtualized Multi-LoRA Framework for Unified LLM Fine-tuning and Serving (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.