Research map / Continual learning / Continual fine-tuning of LLMs

Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Models

ICML 2024 · Continual learning › Continual fine-tuning of LLMs · approach: Continual pretraining of language models · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Model Tailor, a post-training, sparse-mask method that keeps pre-trained MLLM weights and replaces about 10% of fine-tuned parameters (chosen by fusing salience and OBS-style sensitivity scores, with Hessian-based compensation) to mitigate catastrophic forgetting in InstructBLIP and LLaVA-1.5.

Problem

Fine-tuning multi-modal LLMs on new target tasks degrades their performance on original pre-trained tasks (catastrophic forgetting), and full fine-tuning and LoRA do not solve this well.

Approach

Select a sparse subset of fine-tuned parameters (the "model patch") using a hybrid of salience (|sft - pre|) and second-order OBS sensitivity scores, then compensate the selected parameters with Hessian-inverse-based updates (the "patch decorator") while reverting all other parameters to pre-trained values.

Key results

• On InstructBLIP with Flickr30k as target at 10% sparsity, Model Tailor reaches H-score 93.67 vs 92.12 for standard fine-tuning and 92.78 for Grafting.
• On LLaVA-1.5 with Flickr30k target at 10% sparsity, Model Tailor reaches H-score 66.94 vs 63.40 for fine-tuning and 60.03 for Grafting.
• Applied after LoRA fine-tuning on LLaVA-1.5 (Flickr30k), Model Tailor improves the target score by 18.9 points and H-score by 9.9%.

Measured effects reported in the paper

Surprising findings

• Standard fine-tuning of LLaVA-1.5 on Flickr30k drops VizWiz from 50.0 to 27.24, showing severe forgetting even for a strong instruction-tuned MLLM.
• DARE fails to sustain target-task performance on LLaVA-1.5 for OKVQA, dropping back to near zero-shot levels.

Limitations

• Results are reported only for two 7B MLLMs (InstructBLIP and LLaVA-1.5) on captioning and VQA tasks.
• Requires computing Hessian-based statistics (approximated via SparseGPT), and the sparsity and salience/sensitivity mixing weight must be tuned.
• Uses a parameter-distance approximation to the pre-training loss constraint rather than the actual pre-training data.

Experimental setup

Datasets: Flickr30k GQA OKVQA COCO Caption NoCaps AOKVQA VQAv2 VizWiz ScienceQA TextVQA POPE MM-Bench

Benchmarks: COCO Caption NoCaps OKVQA AOKVQA VQAv2 GQA Flickr30k VizWiz ScienceQA TextVQA POPE MM-Bench

Metrics: average performance H-score (harmonic mean of original and target task averages)

Baselines: Standard fine-tuning Model Grafting DARE

Builds on: Optimal Brain Surgeon Lottery Ticket Hypothesis SparseGPT Model Grafting DARE

Closest papers by meaning

• AdaDARE-gamma: Balancing Stability and Plasticity in Multi-modal LLMs through Efficient Adaptation (CVPR 2025 · similarity 0.91)
• Learn from Downstream and Be Yourself in Multimodal Large Language Models Fine-Tuning (ICML 2025 · similarity 0.89)
• Wings: Learning Multimodal LLMs without Text-only Forgetting (NeurIPS 2024 · similarity 0.87)
• SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning (ICML 2025 · similarity 0.87)
• Estimating and Orthogonalizing Unknown Pre-training Gradients for Continual Fine-tuning of Large Language Models (NeurIPS 2026 · similarity 0.87)
• Cross-Modal Unlearning via Influential Neuron Path Editing in Multimodal Large Language Models (AAAI 2026 · similarity 0.87)