Research map / Continual learning / Continual fine-tuning of LLMs

InduceKV: Fixed-Footprint Continual Adaptation of Multimodal LLMs via Inducing KV Memories

NeurIPS 2026 · Continual learning › Continual fine-tuning of LLMs · approach: Mixture-of-experts for multimodal continual tuning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes InduceKV, which keeps an MLLM frozen and stores budgeted, attention-ready layerwise KV memory entries selected by bilevel inducing-set optimization for continual adaptation.

Problem

Continual adaptation of multimodal LLMs under a bounded deployment footprint, where repeated parameter updates or growing replay stores accumulate adaptation state and cause forgetting.

Approach

Freeze the backbone and store each selected training prefix as a retrieval key plus compact layerwise KV payloads appended to self-attention; select the budgeted memory via a bilevel objective balancing current-task likelihood, anchor retention, and log-det spectral coverage, with a tiny retrieval calibration (temperature and per-layer gates).

Key results

• On UCIT with LLaVA-v1.5-7B backbone, InduceKV reaches 69.82 Avg / 65.31 Last, improving over HiDe-LLaVA (68.94 / 64.19).
• On VQAv2 10-task with the LV protocol, InduceKV-LV reports AP 52.64 and forgetting 1.70 versus CL-MoE AP 51.34.
• Single-factor ablation shows removing retrieval-weighted induction costs up to 4.20 AP on VQAv2 and removing inner calibration costs 2.85 UCIT Avg.

Surprising findings

• Removing the anchor retention term costs little on VQAv2 AP (-0.48) relative to the large drop on UCIT/CoIN, indicating anchor importance is task-dependent.
• The paper notes InduceKV adds prefill and KV-injection overhead at inference, so it is not zero-overhead versus replay-free PEFT.

Limitations

• Adds inference overhead from an extra prefix pass and larger attention during prefill relative to replay-free PEFT.
• Baselines are reported under heterogeneous backbones and protocols; the VQACL default result is not strictly T5-matched.
• Theoretical guarantees rely on convexity and bounded-gradient assumptions for the relaxed surrogate, and the TopB rounding gap is only argued under Lipschitz assumptions.

Experimental setup

Datasets: UCIT CoIN VQAv2 VQACL NExT-QA VQA-RAD PathVQA SLAKE ChartQA Chart-to-Table Chart-to-Text MWP&GPS

Benchmarks: UCIT CoIN VQAv2 10-task VQACL domain-incremental Medicine/Chart/Math LiIT

Metrics: Avg Last MAA BWT AP AF/Forget AvgAcc Relative Gain Forgetting Rate

Baselines: FineTune LwF EWC L2P O-LoRA MoELoRA HiDe-LLaVA CL-MoE QUAD VQACL ER DER

Builds on: HiDe-LLaVA CL-MoE QUAD Adapt-∞ VQACL

Closest papers by meaning

• Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning (ICML 2025 · similarity 0.89)
• Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning (NeurIPS 2026 · similarity 0.89)
• CoIN: A Benchmark of Continual Instruction Tuning for Multimodel Large Language Models (NeurIPS 2024 · similarity 0.88)
• Adapt-$\infty$: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selection (ICLR 2025 · similarity 0.88)
• CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration (ICML 2025 · similarity 0.87)
• AdaDARE-gamma: Balancing Stability and Plasticity in Multi-modal LLMs through Efficient Adaptation (CVPR 2025 · similarity 0.87)