Research map / Architectures and efficiency / Pruning and efficient tuning

Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation

NeurIPS 2026 · Architectures and efficiency › Pruning and efficient tuning · approach: LoRA variants and theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Compares full fine-tuning and LoRA across three tasks and three small LLMs, then proposes MoLF, which routes optimizer-level updates between a dense FFT pathway and LoRA pathways using an EPD score, plus a LoRA-only MoLF-E variant.

Problem

Choosing between full fine-tuning (FFT) and LoRA is task- and model-dependent, and existing adaptive LoRA methods allocate updates only within low-rank parameterizations.

Approach

Treat FFT and LoRA as candidate update pathways that all receive gradients and accumulate AdamW moments, then select one physical update per module per step with a gradient-informed, parameter-normalized preconditioned descent score (EPD) and Top-K sparse AdamW.

Key results

• MoLF scores above or within 1.5 points of the better of FFT/LoRA in all nine settings (e.g., Fact Qwen-1.5B 75.15 vs FFT 74.95).
• MoLF-E with two rank-128 LoRA experts beats AdaLoRA and AdaMix in eight of nine settings, with up to 11.70-point gain on Fact.
• On Meta-Llama-3-8B MATH-500, MoLF reaches 31.53% vs 20.60% for LoRA-128 and 32.93% for FFT.

Surprising findings

• Endpoint Frobenius compressibility is highest for Fact, yet rank-128 truncation costs the most absolute score on Fact, so compressibility alone does not explain task differences.
• Updating both pathways (K=2) sharply lowers Qwen2.5-3B Med accuracy from 60.60% to 31.08%.

Limitations

• Original nine-setting results come from a single training run per method; the paired five-seed CounterFact study shows MoLF-FFT interval including zero.
• Standard MoLF retains dense weights, gradients and Adam states, so it provides no LoRA-like training memory savings.
• EPD is a first-order heuristic whose score depends on learning rate, parameter count and LoRA scaling, and is not rank-invariant.

Experimental setup

Datasets: CounterFact MedMCQA Gretel synthetic Text-to-SQL ODA-Mixture-100k

Benchmarks: CounterFact MedMCQA Gretel Text-to-SQL MATH-500

Metrics: Efficacy Score accuracy exact-match accuracy cap hit rate

Baselines: Full fine-tuning LoRA AdaLoRA AdaMix PFN-based routing MoLF w/ no update routing

Builds on: LoRA AdaLoRA AdaMix RS-LoRA AdamW

Closest papers by meaning

• Loquetier: A Virtualized Multi-LoRA Framework for Unified LLM Fine-tuning and Serving (NeurIPS 2025 · similarity 0.94)
• Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning (NeurIPS 2026 · similarity 0.93)
• Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning (NeurIPS 2025 · similarity 0.93)
• GoRA: Gradient-driven Adaptive Low Rank Adaptation (NeurIPS 2025 · similarity 0.93)
• SeedLoRA: A Fusion Approach to Efficient LLM Fine-Tuning (ICML 2025 · similarity 0.92)
• Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.92)