Research map / Continual learning / Continual fine-tuning of LLMs
How to Teach Large Multimodal Models New Skills
Zhen Zhu, Yiming Gong, Yao Xiao, Yaoyao Liu, Derek Hoiem
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| tune only self-attention projections instead of the full LLM (ablation) | LLaVA-OneVision-7B · Held-out suite (5-task sequence) | held-out forgetting: -23.3 -> -0.6 | Improved |
| tune only MLP Gate&Up (freeze Down) instead of full MLP (ablation) | LLaVA-OneVision-7B · Held-out suite (5-task sequence) | held-out forgetting: -15.7 -> -2.1 | Improved |
| freeze Down projection in MLP tuning (ablation) | LLaVA-OneVision-7B · Target tasks (5-task sequence) | target overall: 15.5 -> 27.1 | Improved |
| also tune the vision encoder (vs LLM only) (ablation) | LLaVA-OneVision-7B · Held-out suite (5-task sequence) | held-out forgetting: -10.8 | Hurt |
| freeze W_O in self-attention tuning (SA Proj. QKV) (ablation) | LLaVA-OneVision-7B · Target tasks (5-task sequence) | target overall: 23.1 -> 14.5 | Hurt |
| tune full model instead of LLM only (ablation) | LLaVA-OneVision-7B · Target tasks (5-task sequence) | target learning: 31.8 -> 29.9 | No clear effect |
| tune self-attention projections instead of full LLM (ablation) | LLaVA-NeXT (LLaMA-3 8B) · Held-out suite (5-task sequence) | held-out forgetting: -35.9 -> -7.7 | Improved |
| tune self-attention projections instead of full LLM (ablation) | Qwen2.5-VL-7B · Held-out suite (5-task sequence) | held-out forgetting: -24.6 -> 0.6 | Improved |
| tune MLP Gate&Up instead of full MLP (ablation) | Qwen2.5-VL-7B · Target tasks (5-task sequence) | target overall: 13.9 -> 17.1 | Improved |
| tune MLP Gate&Up instead of full MLP (ablation) | LLaVA-NeXT (LLaMA-3 8B) · Held-out suite (5-task sequence) | held-out forgetting: -16.3 -> -8.7 | Improved |
| tune self-attention projections instead of full model (single task) (ablation) | LLaVA-OneVision-7B · Held-out average (single-task fine-tuning) | held-out change: -3.2 -> +0.0 | Improved |
| tune MLP instead of full LLM (single task) (ablation) | LLaVA-OneVision-7B · Held-out average (single-task fine-tuning) | held-out change: -2.8 -> -1.4 | Improved |
Closest papers by meaning
- Re-Imagining Multimodal Instruction Tuning: A Representation View (ICLR 2025 · similarity 0.88)
- Adapt-$\infty$: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selection (ICLR 2025 · similarity 0.88)
- Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting (ICML 2025 · similarity 0.88)
- SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning (ICML 2025 · similarity 0.87)
- Beyond task performance: evaluating and reducing the flaws of large multimodal models with in-context-learning (ICLR 2024 · similarity 0.87)
- Wings: Learning Multimodal LLMs without Text-only Forgetting (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
