Research map / Large language models / Instruction and fine-tuning

C-LoRA: Contextual Low-Rank Adaptation for Uncertainty Estimation in Large Language Models

NeurIPS 2025 · Large language models › Instruction and fine-tuning · approach: Low-rank adaptation variants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes C-LoRA, a Bayesian LoRA fine-tuning method whose low-rank middle matrix E is sampled from an input-conditioned Gaussian posterior produced by lightweight contextual modules, improving calibration on LLaMA2-7B few-shot commonsense reasoning tasks.

Problem

LoRA fine-tuning of LLMs in data-scarce few-shot settings yields overconfident predictions, and existing uncertainty-aware LoRA methods ignore how input characteristics affect predictive (aleatoric/data) uncertainty.

Approach

Insert a small r×r matrix E between LoRA factors B and A, make E's Gaussian variational posterior conditioned on each input via per-layer contextual MLPs, and train with an amortized ELBO (BLoB-style mean-field VI with input-dependent posterior), so only E is stochastic and the cost is independent of hidden dimension d.

Key results

• C-LoRA (M=10) reaches ECE 6.86% on WG-S and 8.83% on ARC-C, lower than the baselines' best in those tasks' calibration comparisons, while accuracy stays within 1-2% of the best methods (e.g., ARC-C 67.79%).
• After temperature scaling, C-LoRA (M=0, tmp) achieves ECE 4.00% on WG-S and 3.86% on ARC-E, best or second-best in 4 of 6 datasets.
• Under OBQA-to-OOD shift, C-LoRA (M=10) obtains the lowest NLL on Chem (1.31) and Phy (1.42) among the compared methods.

Measured effects reported in the paper

Surprising findings

• Temperature scaling can worsen ECE for C-LoRA (M=10) on some datasets (e.g., OBQA from 4.00% to 6.27%).
• C-LoRA trades a small amount of accuracy for better calibration, and in the OOD Chem setting it has subpar accuracy relative to baselines.

Limitations

• Experiments limited to LLaMA2-7B; scaling to larger models untested.
• No theoretical guarantee that the method disentangles aleatoric from epistemic uncertainty; variational Gaussian posterior is mode-seeking.
• Accuracy is slightly reduced relative to non-Bayesian baselines, a trade-off that is task-dependent.

Experimental setup

Datasets: WinoGrande-small WinoGrande-medium ARC-Challenge ARC-Easy OpenBookQA BoolQ Chem Phy MMLU

Benchmarks: WinoGrande-small WinoGrande-medium ARC-Challenge ARC-Easy OpenBookQA BoolQ Chem Phy

Metrics: ACC ECE NLL

Baselines: MAP Monte Carlo dropout Deep Ensemble Laplace-LoRA BLoB

Builds on: LoRA BLoB Laplace-LoRA Deep Ensemble Monte Carlo dropout

Closest papers by meaning

• Bayesian Low-rank Adaptation for Large Language Models (ICLR 2024 · similarity 0.93)
• QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models (ICLR 2024 · similarity 0.92)
• Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.91)
• Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment (ICML 2025 · similarity 0.91)
• Dynamic Low-Rank Sparse Adaptation for Large Language Models (ICLR 2025 · similarity 0.91)
• Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation (NeurIPS 2026 · similarity 0.91)