Research map / Large language models / Instruction and fine-tuning
qa-FLoRA: Data-free query-adaptive Fusion of LoRAs for LLMs
Shreya Shukla, Aditya Sriram, Milinda Kuppur Narayanaswamy, Hiteshi Jain
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace static equal-weight fusion with qa-FLoRA | LoRA fusion · Average across 3 domains | score: 20.4 -> 25.8 (LLaMA-2); 38.5 -> 44.2 (LLaMA-3) | Improved |
| replace centroid similarity with KL-divergence weights | LoRA fusion · Average across 3 domains | score: 18.8 -> 25.8 (LLaMA-2); 34.4 -> 44.2 (LLaMA-3) | Improved |
| compute KL on last token rather than averaging all query tokens (ablation) | qa-FLoRA · Average across 3 domains | score: 23.5 -> 25.8 | Improved |
| qa-FLoRA vs centroid on code | LoRA fusion · HumanEval composite | pass@1: 18.8 -> 17.7 (LLaMA-2); 33.3 -> 34.4 (LLaMA-3) | No clear effect |
| qa-FLoRA vs supervised LoRAFlow | LoRA fusion · Average across 3 domains | score: 30.9 -> 25.8 (LLaMA-2); 46.1 -> 44.2 (LLaMA-3) | Hurt |
| qa-FLoRA vs static fusion on code (LLaMA-2) | LoRA fusion · HumanEval composite | pass@1: 17.7 -> 17.7 | No clear effect |
| qa-FLoRA vs LoRAHub | LoRA fusion · Average across 3 domains (LLaMA-2) | score: 26.4 -> 25.8 | Hurt |
Closest papers by meaning
- Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation (NeurIPS 2026 · similarity 0.89)
- QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models (ICLR 2024 · similarity 0.89)
- LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades (NeurIPS 2025 · similarity 0.89)
- Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering (ICLR 2025 · similarity 0.89)
- Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment (ICML 2025 · similarity 0.88)
- Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
