Research map / Architectures and efficiency / Pruning and efficient tuning
The Appeal and Reality of Recycling LoRAs with Adaptive Merging
Haokun Liu, Gyung Hyun Je, Marco Ciccone, Zhenlin Xu, Prasanth YSS, Colin Raffel
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add target-task LoRA to the merging pool (ablation) | LoraHub (30) · 62 tasks (Llama 3.1 8B) | average accuracy: 0.563 -> 0.663 | Improved |
| add target-task LoRA to the merging pool (ablation) | Arrow (30) · 62 tasks (Llama 3.1 8B) | average accuracy: 0.471 -> 0.613 | Improved |
| add target-task LoRA to the merging pool (ablation) | pi-Tuning (20) · 62 tasks (Llama 3.1 8B) | average accuracy: 0.668 -> 0.668 | No clear effect |
| add target-task LoRA to the merging pool (ablation) | Ours (30) · 62 tasks (Llama 3.1 8B) | average accuracy: 0.652 -> 0.675 | Improved |
| use adaptive merging instead of target-task LoRA alone | Ours (30) · 62 tasks (Llama 3.1 8B), no target LoRA in pool | average accuracy: 0.657 -> 0.652 | No clear effect |
| use evaluation-based instead of random LoRA selection (non-adaptive, Simple avg.) (ablation) | Simple avg. (30) · 62 tasks, without target-task LoRA | average accuracy: 0.496 -> 0.543 | Improved |
| use evaluation-based instead of random LoRA selection when target-task LoRA is in pool (ablation) | adaptive merging · 62 tasks | average accuracy | No clear effect |
| replace recycled LoRAs with randomly reinitialized LoRAs (target LoRA kept) (ablation) | Ours · 62 tasks, k in 5,10,20,30 | average accuracy | No clear effect |
| use gradient-based instead of gradient-free coefficient tuning (ablation) | model-level merging · 20 ablation tasks, k=30, no target LoRA | average accuracy: 0.590 -> 0.607 | Improved |
| apply standard regularizers (dropout, weight decay, batch size) to target-task LoRA training (ablation) | target-task LoRA · 62 tasks | average accuracy: 0.672 | Hurt |
| use in-house target-task LoRAs instead of recycled hub LoRAs as pool | Ours with target LoRA · 62 tasks | average accuracy | Improved |
| omit top-ranked in-house LoRAs from the pool (m in 5,10,15,20) (ablation) | Ours, k=10 · 62 tasks | average accuracy | Hurt |
Closest papers by meaning
- Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering (ICLR 2025 · similarity 0.90)
- IterIS: Iterative Inference-Solving Alignment for LoRA Merging (CVPR 2025 · similarity 0.90)
- LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades (NeurIPS 2025 · similarity 0.89)
- LoRA Recycle: Unlocking Tuning-Free Few-Shot Adaptability in Visual Foundation Models by Recycling Pre-Tuned LoRAs (CVPR 2025 · similarity 0.89)
- Accurate and Efficient Low-Rank Model Merging in Core Space (NeurIPS 2025 · similarity 0.87)
- Model Merging in Pre-training of Large Language Models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
