Research map / Continual learning / Continual fine-tuning of LLMs

SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning

ICML 2025 · Continual learning › Continual fine-tuning of LLMs · approach: Mixture-of-experts for multimodal continual tuning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SEFE for multimodal continual instruction tuning, combining Answer Style Diversification (ASD) data reformatting against superficial forgetting with RegLoRA, a regularization on top-magnitude elements of past LoRA updates against essential forgetting; evaluated on CoIN and a new CoIN-ASD.

Problem

Multimodal continual instruction tuning of MLLMs suffers from catastrophic forgetting, which the authors split into superficial forgetting (answer-format drift caused by later tasks' answer styles) and essential forgetting (true knowledge loss), with the former obscuring measurement of the latter.

Approach

Diversify answer styles within each task (yes/no, multiple-choice, short, brief, detailed) so the model is not biased to one format, and regularize the top-M% elements of past LoRA weight updates (ΔW = BA) when training new LoRAs to preserve prior knowledge.

Key results

• On CoIN-ASD under TA, SEFE reaches MFN 58.57, MAA 63.04 and BWT -10.45, versus LoRA baseline 41.59 / 39.53 / -28.62.
• Adding ASD alone to LoRA raises MFN from 41.59 to 47.88 and BWT from -28.62 to -20.26 (TA).
• Under KC metrics, SEFE achieves MFN 73.29 and MAA 75.61, outperforming O-LoRA+ASD (71.40 / 75.07).

Measured effects reported in the paper

Surprising findings

• Converting only 10% of each task's data into alternative answer styles substantially reduces superficial forgetting.
• Regularizing the top 2% of weight update elements (ΔW) outperforms regularizing A or B matrices.

Limitations

• Evaluation relies on exact-match TA and an LLM judge (Qwen1.5-32B) for KC, which the authors note has limits in extreme format cases.
• ASD transformations depend on MLLM-generated distractors and rewrites, and some tasks need manual adjustments.
• Experiments are limited to the single CoIN benchmark sequence with LLaVA-1.5-7B.

Experimental setup

Datasets: CoIN ScienceQA TextVQA ImageNet GQA VizWiz VQAv2 OCR-VQA

Benchmarks: CoIN CoIN-ASD

Metrics: MFT MFN MAA BWT accuracy (TA) knowledge capability (KC)

Baselines: FFT LoRA O-LoRA LoTA

Builds on: LoRA O-LoRA CoIN

Closest papers by meaning

• CoIN: A Benchmark of Continual Instruction Tuning for Multimodel Large Language Models (NeurIPS 2024 · similarity 0.90)
• Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning (NeurIPS 2026 · similarity 0.89)
• Estimating and Orthogonalizing Unknown Pre-training Gradients for Continual Fine-tuning of Large Language Models (NeurIPS 2026 · similarity 0.88)
• Large Continual Instruction Assistant (ICML 2025 · similarity 0.88)
• Unlocking the Power of Function Vectors for Characterizing and Mitigating Catastrophic Forgetting in Continual Instruction Tuning (ICLR 2025 · similarity 0.88)
• Learn more, but bother less: parameter efficient continual learning (NeurIPS 2024 · similarity 0.88)