Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes

Training-Free Bayesianization for Low-Rank Adapters of Large Language Models

NeurIPS 2025 · Probabilistic and scientific modeling › Uncertainty and Gaussian processes · approach: Priors and functional variational uncertainty · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Training-Free Bayesianization (TFB), which converts trained LoRA adapters into Bayesian ones by binary-searching the largest isotropic low-rank Gaussian noise scale that keeps anchor-set NLL within a tolerance, with a theorem tying this to generalized variational inference.

Problem

Uncertainty estimation for LLMs fine-tuned with LoRA; existing Bayesian LoRA methods (BLoB, Laplace-LoRA) need costly retraining, continued training, or gradient-based post-training.

Approach

Project a full-rank isotropic Gaussian onto the LoRA low-rank subspace via an SVD regrouping of B and A, then choose the noise scale sigma_q as the largest value whose anchor-set performance change stays within a tolerance found by binary search; theoretically this equals KL-regularized generalized variational inference.

Key results

• On Llama3.1-8B, BLoB-Mean+TFB reduces ECE on WG-M from 9.37% to 3.83%, and MLE+TFB reduces ECE on ARC-E from 7.00% to 5.14%.
• MLE+TFB ECE on combined commonsense data drops from 4.74% to 1.05% on Llama3.1-8B, per the backbone analysis.
• TFB runs up to 5x faster than BLoB and uses about half the GPU memory on the larger BoolQ dataset, though it is slower on small datasets like WG-S.

Measured effects reported in the paper

Surprising findings

• On small datasets like WG-S, TFB is about 3x slower than BLoB because of the binary search, so its efficiency advantage depends on dataset size.
• Constant low-rank standard deviation (C-STD) gets better NLL than the final TFB but at a large cost in accuracy.

Limitations

• Requires an in-distribution anchor dataset to set the noise scale, and its quality affects results.
• Constraining posteriors to low-rank isotropic Gaussians may miss more complex uncertainty structure.
• Evaluated only on classification-style commonsense tasks; performance on complex generation tasks is untested.

Experimental setup

Datasets: WinoGrande-S WinoGrande-M ARC-Challenge ARC-Easy OpenBookQA BoolQ MMLU college chemistry MMLU college physics

Benchmarks: WinoGrande-S WinoGrande-M ARC-C ARC-E OBQA BoolQ MMLU Chemistry (OOD) MMLU Physics (OOD)

Metrics: accuracy ECE NLL

Baselines: MCD Deep Ensemble MonteCLoRA BLoB Laplace-LoRA MLE MAP

Builds on: BLoB LoRA Laplace-LoRA generalized variational inference

Closest papers by meaning

• BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models (NeurIPS 2024 · similarity 0.94)
• Bayesian Low-rank Adaptation for Large Language Models (ICLR 2024 · similarity 0.91)
• C-LoRA: Contextual Low-Rank Adaptation for Uncertainty Estimation in Large Language Models (NeurIPS 2025 · similarity 0.90)
• Dynamic Low-Rank Sparse Adaptation for Large Language Models (ICLR 2025 · similarity 0.88)
• Large Language Models to Enhance Bayesian Optimization (ICLR 2024 · similarity 0.88)
• QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models (ICLR 2024 · similarity 0.88)