Research map / Architectures and efficiency / Pruning and efficient tuning

On the Construction and Implications of Low-Loss Valleys in LoRA-based Bayesian Inference

NeurIPS 2026 · Architectures and efficiency › Pruning and efficient tuning · approach: LoRA variants and theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds segmented Bézier curves in LoRA parameter space (free and anchored variants) to connect LoRA optima through low-loss valleys and uses uniform curve averaging plus a JSD regularizer for Bayesian model averaging with higher mutual information on Qwen2.5-7B.

Problem

Epistemic uncertainty estimation for LoRA-fine-tuned LLMs is hard; recent results suggest multi-mode methods such as deep ensembles offer little benefit over single-mode methods in the LoRA regime, and it is unknown whether low-loss continuous structure exists in LoRA space.

Approach

Apply segmented Bézier mode-connectivity curves directly in the LoRA (A,B) parameter space, either jointly optimized from scratch (FLC) or anchored to independently fine-tuned LoRA optima (ALC), average predictions uniformly along the curve for BMA, and add a clipped JSD regularizer to increase functional diversity.

Key results

• On ARC-Challenge, ALC(9,2) reaches 0.908 accuracy versus 0.889 for MAP and 0.902 for DE(9), with ALC(9,2) and FLC(9,2) giving the highest mutual information among methods on most datasets (e.g., 0.074 vs 0.051 for DE(9) on ARC-Challenge).
• Linear interpolation between anchors shows loss barriers at midpoints, while ALC and FLC curves traverse low-loss valleys on training data.
• Reported FLC shows a generalization gap: it does not consistently match ALC or DE on test performance, notably on winogrande_m.

Surprising findings

• Uniform averaging along the curve (temperature T to infinity) outperforms standard posterior weighting with T=1, which the authors attribute to double-counting training data.
• Jointly optimized free curves (FLC) do not consistently generalize better than anchored curves despite equal representational capacity.

Limitations

• Restricted to one-dimensional curves; higher-dimensional surfaces or per-adapter curves were not pursued.
• FLC shows an inconsistent generalization gap relative to ALC and DE.
• Uniform weighting is an empirical choice motivated by a double-counting argument and may not suffice in higher dimensions.

Experimental setup

Datasets: ARC-Challenge ARC-Easy OBQA BoolQ Winogrande-S Winogrande-M

Benchmarks: ARC-Challenge ARC-Easy OBQA BoolQ Winogrande-S Winogrande-M

Metrics: accuracy expected calibration error log-likelihood mutual information

Baselines: MAP BLoB ScalaBL Deep Ensemble Linear mode connectivity Lin(3)

Builds on: LoRA Flat-LoRA mode connectivity via Bezier curves deep ensembles BLoB

Closest papers by meaning

• LoRA Training Provably Converges to a Low-Rank Global Minimum Or It Fails Loudly (But it Probably Won't Fail) (ICML 2025 · similarity 0.89)
• Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape (ICML 2025 · similarity 0.89)
• Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment (ICML 2025 · similarity 0.89)
• Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.88)
• C-LoRA: Contextual Low-Rank Adaptation for Uncertainty Estimation in Large Language Models (NeurIPS 2025 · similarity 0.88)
• The Expressive Power of Low-Rank Adaptation (ICLR 2024 · similarity 0.88)