Research map / Large language models / Instruction and fine-tuning

Hybrid-LoRA: Bridging Full Fine-Tuning and Low-Rank Adaptation for Post-Training

NeurIPS 2026 · Large language models › Instruction and fine-tuning · approach: Low-rank adaptation variants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Hybrid-LoRA, which uses a LoRA probing stage and a gradient-weighted Hybrid-Score to pick ~10% of linear modules for full fine-tuning and LoRA-adapts the rest for RLVR post-training (GRPO/GSPO) of Qwen-2.5 models.

Problem

PEFT methods such as LoRA leave a performance gap to full fine-tuning in RLVR post-training for complex reasoning, while full fine-tuning is memory- and compute-heavy.

Approach

Score each candidate linear module by the mean-times-std of first-order gradient-weighted sensitivity of its LoRA SVD coefficients under random co-active masking, then assign the lowest-scoring modules (under a 10% budget) to full fine-tuning and LoRA the rest.

Key results

• Hybrid-LoRA with GSPO on Qwen-2.5 1.5B reaches 81.6 Math-500 pass@1 vs 82.0 for full fine-tuning, using ~105M vs 1544M tunable parameters.
• Hybrid-LoRA on Qwen-2.5 7B with GSPO reaches 94.4 Math-500, 60.0 AIME-24, and 77.9 GPQA, outperforming the AutoLoRA baseline.
• The paper reports an average 4.36% relative improvement and up to 5.65% over the best PEFT baseline across benchmarks.

Surprising findings

• Simply increasing the number of PEFT trainable parameters does not close the gap to full fine-tuning; module allocation matters more.
• Selected full fine-tuned modules concentrate in later-layer attention projections (q, k, v, o), while FFN modules are mostly LoRA-adapted.

Limitations

• Experiments only scale up to 7B models.
• Two-stage probing-then-training pipeline is static; no end-to-end or dynamic allocation.
• Baseline comparisons are inconsistent across scales (e.g., not all baselines reported at 3B and 7B).

Experimental setup

Datasets: Mixture-of-Thoughts Math-500 AIME-24 AIME-25 MMLU-Pro GPQA LeetCodeDataset

Benchmarks: Math-500 AIME-24 AIME-25 MMLU-Pro GPQA LeetCodeDataset

Metrics: pass@1 pass@4 accuracy

Baselines: IO prompting CoT prompting Full-FT LoRA-FFN AdaLoRA AutoLoRA LoRA-Drop

Builds on: LoRA AdaLoRA AutoLoRA LoRA-Drop GRPO GSPO

Closest papers by meaning

• Tina: Tiny Reasoning Models via LoRA (ICLR 2026 · similarity 0.92)
• Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning (NeurIPS 2026 · similarity 0.90)
• Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation (NeurIPS 2026 · similarity 0.90)
• SeedLoRA: A Fusion Approach to Efficient LLM Fine-Tuning (ICML 2025 · similarity 0.89)
• Loquetier: A Virtualized Multi-LoRA Framework for Unified LLM Fine-tuning and Serving (NeurIPS 2025 · similarity 0.89)
• First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-Training (NeurIPS 2025 · similarity 0.89)