Research map / Continual learning / Continual fine-tuning of LLMs
AdaDARE-gamma: Balancing Stability and Plasticity in Multi-modal LLMs through Efficient Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AdaDARE-γ, a training-free fusion method that adaptively drops fine-tuned delta parameters (Hessian-guided probabilities) and scales them by a γ factor to balance catastrophic forgetting and target-task plasticity in MLLMs.
Problem
Adapting multimodal LLMs to new target tasks via fine-tuning causes catastrophic forgetting of pre-trained task performance, creating a stability-plasticity trade-off.
Approach
Extends DARE with parameter-wise adaptive drop probabilities derived from a Hessian-based optimality criterion, and adds a scalar γ that controls how much of the selected task-specific delta is injected into pre-trained weights.
Key results
- On InstructBLIP (Flickr30k target), AdaDARE-γ reaches H-score 96.05 versus 92.28 for full fine-tuning and 94.69 for Model-Tailor, with target Flickr30k 99.7 vs 101.3 for fine-tuning.
- On LLaVA-1.5 (OKVQA target), AdaDARE-γ achieves H-score 60.74 and target 57.21 vs 55.72 for fine-tuning, while Model-Tailor gets 59.25.
- The paper claims AdaDARE-γ keeps 98.2% of pre-training effectiveness and 98.7% of fine-tuning target performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AdaDARE-γ instead of Model-Tailor | InstructBLIP fine-tuned on Flickr30k · Flickr30k fine-tuning trade-off | H-score: 94.69 -> 96.05 | Helped |
| Use AdaDARE-γ instead of Model-Tailor | InstructBLIP fine-tuned on GQA · GQA fine-tuning trade-off | H-score: 69.47 -> 71.30 | Helped |
| Use AdaDARE-γ instead of Model-Tailor | LLaVA-1.5 fine-tuned on Flickr30k · Flickr30k fine-tuning trade-off | H-score: 75.58 -> 75.89 | Helped |
| Use AdaDARE-γ instead of Model-Tailor | LLaVA-1.5 fine-tuned on OKVQA · OKVQA fine-tuning trade-off | H-score: 59.25 -> 60.74 | Helped |
| Combine AdaDARE-γ with LoRA instead of using Model-Tailor | LLaVA-1.5 fine-tuned on Flickr30k with LoRA · Flickr30k fine-tuning trade-off | H-score: 72.67 -> 73.73 | Helped |
| Combine AdaDARE-γ with LoRA instead of using Model-Tailor | LLaVA-1.5 fine-tuned on OKVQA with LoRA · OKVQA fine-tuning trade-off | H-score: 61.21 -> 61.41 | Helped |
| Replace random parameter selection with adaptive parameter selection (ablation) | AdaDARE-γ · Flickr30k across drop rates | target task performance | Helped |
| Vary the drop rate (ablation) | AdaDARE-γ · Flickr30k | target task performance | Mixed |
| Vary γ within [0.5, 0.8] (ablation) | AdaDARE-γ · InstructBLIP (Flickr30k, GQA) and LLaVA-1.5 (Flickr30k, OKVQA) | performance | No clear effect |
Surprising findings
- Plain DARE reaches nearly full fine-tuning target performance but degrades sharply on pre-trained tasks, indicating drop-and-rescale alone does not preserve multimodal capabilities.
- Fine-tuning only the last 12 LLM layers of LLaVA-1.5 gave much lower performance than PEFT methods, which the authors attribute to limited GPU resources.
Limitations
- Relies on manual tuning of γ; theoretical bounds may not align with optimal practical values.
- Evaluated only on InstructBLIP and LLaVA-1.5 with two target tasks each (captioning and VQA).
- Hessian computation adds overhead compared with DARE, and the authors use a sparsity ratio fixed at 90%.
Experimental setup
Datasets: Flickr30k GQA OKVQA
Benchmarks: COCO Caption NoCaps OKVQA AOKVQA VQAv2 GQA Flickr30k VizWiz SQA TextVQA POPE MM-Bench
Metrics: Avg H-score CIDEr accuracy
Baselines: Fine-tuning DARE Model-Tailor LoRA Zero-shot
Builds on: DARE Task Arithmetic Model-Tailor LoRA
Closest papers by meaning
- Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Models (ICML 2024 · similarity 0.91)
- Learn from Downstream and Be Yourself in Multimodal Large Language Models Fine-Tuning (ICML 2025 · similarity 0.88)
- SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning (ICML 2025 · similarity 0.88)
- Wings: Learning Multimodal LLMs without Text-only Forgetting (NeurIPS 2024 · similarity 0.88)
- Estimating and Orthogonalizing Unknown Pre-training Gradients for Continual Fine-tuning of Large Language Models (NeurIPS 2026 · similarity 0.87)
- Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.