Research map / Continual learning / Continual fine-tuning of LLMs
LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoRASculpt, which sparsifies LoRA updates with theoretical sparsity bounds and adds a pretrained-magnitude-guided Conflict Mitigation Regularizer to fine-tune LLaVA-1.5 while retaining general knowledge.
Problem
LoRA fine-tuning of MLLMs on downstream tasks (VQA, captioning) introduces harmful redundant parameters and optimization conflicts with pretrained knowledge, causing catastrophic forgetting of general abilities and sometimes hurting target performance.
Approach
Introduce magnitude-based sparse masks into both LoRA factors A and B, with theoretical bounds showing the product BA stays sparse at high sparsity, plus a regularizer penalizing BA entries where pretrained weights have high magnitude, steering updates into less critical pretrained parameter locations.
Key results
- At LoRA rank 32 on IconQA, LoRASculpt reaches 56.10 average source and 85.26 target accuracy versus 44.79 source and 82.52 target for LoRA (ablation Table 3 reports 70.68 avg vs 63.65).
- On COCO-Caption at rank 64, LoRASculpt achieves 120.39 target score vs 107.90 for LoRA while keeping source at 54.84 vs 39.20 average.
- Applying the method to the connector (Ours+Ours) raises IconQA average to 72.45 vs 70.04 with FFT connector in Table 2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add sparsifying for redundancy reduction (SRR) to LoRA (ablation) | LLaVA-1.5 · IconQA | Source average: 44.79 -> 54.73 | Helped |
| add sparsifying for redundancy reduction (SRR) to LoRA (ablation) | LLaVA-1.5 · IconQA | Target accuracy: 82.52 -> 84.45 | Helped |
| add regularizing for knowledge harmonization (RKH) to the SRR method (ablation) | LoRASculpt · IconQA | Source average: 54.73 -> 56.10 | Helped |
| add regularizing for knowledge harmonization (RKH) to the SRR method (ablation) | LoRASculpt · IconQA | Target accuracy: 84.45 -> 85.26 | Helped |
| add sparsifying for redundancy reduction (SRR) to LoRA (ablation) | LLaVA-1.5 · COCO-Caption | Source average: 49.86 -> 53.97 | Helped |
| add sparsifying for redundancy reduction (SRR) to LoRA (ablation) | LLaVA-1.5 · COCO-Caption | CIDEr: 110.27 -> 120.11 | Helped |
| add regularizing for knowledge harmonization (RKH) to the SRR method (ablation) | LoRASculpt · COCO-Caption | Source average: 53.97 -> 54.25 | Helped |
| add regularizing for knowledge harmonization (RKH) to the SRR method (ablation) | LoRASculpt · COCO-Caption | CIDEr: 120.11 -> 120.35 | Helped |
| use LoRASculpt instead of Model Tailor | LLaVA-1.5, rank 32 · IconQA | Source average: 50.97 -> 56.10 | Helped |
| use LoRASculpt instead of Model Tailor | LLaVA-1.5, rank 32 · IconQA | Target accuracy: 81.76 -> 85.26 | Helped |
| use LoRASculpt instead of Model Tailor | LLaVA-1.5, rank 32 · COCO-Caption | CIDEr: 117.64 -> 120.35 | Helped |
| use LoRASculpt instead of Model Tailor | LLaVA-1.5, rank 64 · IconQA | Source average: 46.80 -> 55.34 | Helped |
Surprising findings
- Simply magnitude-pruning trained LoRA improves both general and downstream performance, indicating substantial harmful redundancy.
- LoRA at rank 64 on IconQA collapses to near-zero accuracy on upstream tasks, largely attributed to forgetting in the MLLM connector.
Limitations
- Experiments limited to two downstream tasks (IconQA, COCO-Caption) with 10k training samples each and one MLLM family (LLaVA-1.5).
- Importance is measured by parameter magnitude only, without gradient information, and the sparse mask is fixed after one-shot pruning.
Experimental setup
Datasets: IconQA COCO-Caption OKVQA OCRVQA GQA TextVQA
Benchmarks: IconQA COCO-Caption OKVQA OCRVQA GQA TextVQA
Metrics: accuracy source (upstream) average target (downstream) performance CIDEr
Baselines: LoRA DoRA L2-Regularization Orth-Reg DARE Model Tailor
Builds on: LoRA LLaVA-1.5 Model Tailor DARE DoRA
Closest papers by meaning
- Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning (ICML 2025 · similarity 0.90)
- CoIN: A Benchmark of Continual Instruction Tuning for Multimodel Large Language Models (NeurIPS 2024 · similarity 0.90)
- LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades (NeurIPS 2025 · similarity 0.89)
- Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation (NeurIPS 2026 · similarity 0.89)
- Unified Knowledge Maintenance Pruning and Progressive Recovery with Weight Recalling for Large Vision-Language Models (AAAI 2025 · similarity 0.89)
- Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.