Research map / Vision-language and multimodal / CLIP and zero-shot
Learn from Downstream and Be Yourself in Multimodal Large Language Models Fine-Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SPIDER, which ranks parameters by frozen pre-trained weight magnitude versus accumulated fine-tuning gradient and updates only the downstream-important ones via an importance-aware mask, to reduce catastrophic forgetting when fine-tuning MLLMs.
Problem
Fine-tuning multimodal LLMs on downstream tasks causes forgetting of pre-trained generalization abilities; the trade-off between specialization and generalization is unresolved.
Approach
Measure per-parameter importance for generic knowledge (pre-trained weight magnitude) and specialized knowledge (accumulated fine-tuning gradient), then update only parameters whose specialization importance exceeds generalization importance, with soft importance-weighted mask rescaling and restoring the rest to pre-trained values.
Key results
- On Flickr30k with VILA1.5-3B, SPIDER reaches H-Average 66.61 vs 55.16 for Full FT and 59.52 for Half FT.
- On COCO-Caption with LLaVA-1.5-7B, SPIDER reaches target CIDEr 114.74 and H-Average 73.89 vs 63.80 for Full FT.
- On ScienceQA with VILA1.5-3B, SPIDER reaches H-Average 66.60 vs 61.82 for Full FT, while on IconQA it attains H-Average 66.71 vs 56.13.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SPIDER instead of the strongest listed baseline, Half FT | VILA1.5-3B · Flickr30k | H-Average: 59.52 -> 66.61 | Helped |
| Use SPIDER instead of the strongest listed baseline, Half FT | VILA1.5-3B · COCO-Capation | H-Average: 68.13 -> 76.86 | Helped |
| Use SPIDER instead of the strongest listed baseline, Half FT | LLaVA-1.5-7B · Flickr30k | H-Average: 58.12 -> 66.93 | Helped |
| Use SPIDER instead of Full FT | LLaVA-1.5-7B · COCO-Capation | H-Average: 63.80 -> 73.89 | Helped |
| Use SPIDER instead of Half FT | VILA1.5-3B · IconQA | H-Average: 63.89 -> 66.71 | Helped |
| Use SPIDER instead of Half FT | VILA1.5-3B · ScienceQA | H-Average: 64.86 -> 66.60 | Helped |
| Select candidates randomly rather than using SPIDER's importance discrepancy criterion (ablation) | SPIDER · Flickr30k and COCO-Capation | H-Average: 59.52 vs 61.35; 68.13 vs 70.60 | Hurt |
| Select candidates by magnitude alone rather than using SPIDER's importance discrepancy criterion (ablation) | SPIDER · Flickr30k and COCO-Capation | H-Average: 60.08 vs 61.35; 68.95 vs 70.60 | Hurt |
| Select candidates by gradient alone rather than using SPIDER's importance discrepancy criterion (ablation) | SPIDER · Flickr30k and COCO-Capation | H-Average: 55.46 vs 61.35; 65.93 vs 70.60 | Hurt |
| Add ISM to IDM (ablation) | SPIDER · Flickr30k | H-Average: 61.35 -> 66.61 | Helped |
| Add ISM to IDM (ablation) | SPIDER · COCO-Capation | H-Average: 70.60 -> 76.86 | Helped |
| Extend fine-tuning from 5 to 10 epochs (ablation) | Full FT, VILA1.5-3B · Flickr30k | Source performance: 47.04 -> 44.08 | Hurt |
Surprising findings
- Solely selecting parameters by low pre-trained magnitude (generalization importance) significantly limits downstream adaptation, and selecting by gradient alone undermines generalization.
- Full fine-tuning on the VILA architecture reduces source performance on captioning, while on VQA it improves target accuracy at the cost of generalization.
Limitations
- Requires storing pre-trained weights and gradient information, adding memory usage linear in trainable parameters.
- Benefit diminishes when downstream task closely aligns with the pre-training distribution, where fewer updates are needed.
Experimental setup
Datasets: Flickr30k COCO-Caption IconQA ScienceQA OKVQA TextVQA GQA OCRVQA
Benchmarks: Flickr30k COCO-Caption IconQA ScienceQA OKVQA TextVQA OCRVQA GQA
Metrics: CIDEr accuracy H-Average O-Average
Baselines: Full FT L2-Reg Grafting Half FT DARE Tailor Zero-shot
Builds on: DARE Tailor Grafting LoRA magnitude pruning
Closest papers by meaning
- Explore How to Inject Beneficial Noise in MLLMs (AAAI 2026 · similarity 0.91)
- Be Confident: Uncovering Overfitting in MLLM Multi-Task Tuning (ICML 2025 · similarity 0.90)
- MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning (ICLR 2025 · similarity 0.90)
- On Path to Multimodal Generalist: General-Level and General-Bench (ICML 2025 · similarity 0.90)
- Accelerating Multimodal Large Language Models by Searching Optimal Vision Token Reduction (CVPR 2025 · similarity 0.89)
- Visual Instruction Bottleneck Tuning (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.