Research map / Continual learning / Continual fine-tuning of LLMs
When Large Multimodal Models Confront Evolving Knowledge: Challenges and Explorations
Kailin Jiang, Yuntao Du, Yukai Ding, Yuchen Ren, Ning Jiang, Zhi Gao, Zilong Zheng, Lei Liu, Bin Li, Qing Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune LMM on evolving knowledge (Full-FT) instead of no injection | LLaVA-v1.5 · MMEvoke (all) | CEM: 4.89 -> 18.02 | Improved |
| use multimodal RAG (UniIR) instead of fine-tuning | LLaVA-v1.5 · MMEvoke (all) | CEM: 18.02 -> 40.68 | Improved |
| provide sufficient context instead of retrieved context | Qwen-VL-Chat · MMEvoke (all) | CEM: 32.75 -> 48.96 | Improved |
| add knowledge-agnostic text augmentation to SFT (ablation) | LLaVA-v1.5 SFT · MMEvoke (all) | CEM: -5.66% CEM, -7.78% F1 (text) | Hurt |
| add knowledge-agnostic image augmentation to SFT (ablation) | LLaVA-v1.5 SFT · MMEvoke Entity subset | CEM: -47.36% CEM, -12.76% F1 | Hurt |
| add knowledge-aware text augmentation to SFT (ablation) | LLaVA-v1.5 SFT · MMEvoke (all) | CEM: +11.32% CEM, +39.82% F1 | Improved |
| add knowledge-aware visual augmentation to SFT (ablation) | LLaVA-v1.5 SFT · MMEvoke Entity subset | CEM: +43.28% CEM, +13.19% F1 | Improved |
| add Replay (10% pretraining data) to LoRA injection (ablation) | LLaVA-v1.5 LoRA · MIA-Bench (retention) | score: 29.66 -> 62.33 | Improved |
| use MoELoRA instead of LoRA (ablation) | LLaVA-v1.5 · MIA-Bench (retention) | score: 29.66 -> 64.97 | Improved |
| add EWC regularization to fine-tuning (ablation) | LLaVA-v1.5 · General capability tests | score: MIA-Bench 25.25 -> 29.79 | No clear effect |
| add LwF to fine-tuning (ablation) | LLaVA-v1.5 · General capability tests | score: MMDU 13.03 -> 13.68 | Mixed |
| add knowledge-aware text augmentation to injection (retention effect) (ablation) | LLaVA-v1.5 · General capability tests | average degradation (ranking): ranked 4 vs Full-FT 8, LoRA 7 | Mixed |
Closest papers by meaning
- Towards Understanding How Knowledge Evolves in Large Vision-Language Models (CVPR 2025 · similarity 0.89)
- Towards Unified Multimodal Editing with Enhanced Knowledge Collaboration (NeurIPS 2024 · similarity 0.87)
- MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models (AAAI 2026 · similarity 0.87)
- Benchmarking Multimodal Knowledge Conflict for Large Multimodal Models (AAAI 2026 · similarity 0.86)
- ComprehendEdit: A Comprehensive Dataset and Evaluation Framework for Multimodal Knowledge Editing (AAAI 2025 · similarity 0.86)
- Diving into Self-Evolving Training for Multimodal Reasoning (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
