Research map / Vision-language and multimodal / Multimodal LLMs

Which Concepts to Forget and How to Refuse? Decomposing Concepts for Continual Unlearning in Large Vision-Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Bias-aware multi-image robustness · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CORE, a concept-aware continual unlearning framework for LVLMs that decomposes forget targets into visual/textual concepts, uses a concept modulator and a mixture of routed refusers, and calibrates refusals at inference.

Problem

Continual unlearning in large vision-language models: sequential deletion requests distort shared representations, causing irrelevant refusals on earlier forget data and over-refusal on retain queries.

Approach

Grounds refusal in fine-grained visual attribute and textual intent concepts decomposed from each forget category; a concept modulator reweights concept activations per category, and a relevance-guided router reuses or adapts a mixture of refusers (connection-module adapters) for new concepts, with inference-time calibration by relevance to unlearned tasks.

Key results

• On the Vicuna-based LVLM after all 16 tasks (Last), CORE reaches 88.02% Answer Rate and 90.67% CRR, versus 52.82% CRR for MoEAdapter and 73.03% for O^3.
• CORE's Last Refusal Gap is 3.74 versus 6.99 for MoEAdapter, the best baseline on that metric (O^3 is 9.32 in Avg).
• On Llama-2, CORE improves Last Answer Rate by 17.68 points and CRR by 7.80 points over the runner-up O^3.

Surprising findings

• The ablation without the concept modulator (ACT and CAL retained) drops Last Answer Rate to 92.51 -> 86.75 only partly, but removing MOD while keeping ACT lowers Last CRR to 71.14 and Last AR to 86.75, showing modulator importance.
• Without relevance-guided activation, refuser usage concentrates on a few refusers, which the authors say hinders maintaining distinct refusal behaviors.

Limitations

• Relies on ChatGPT-generated concept descriptions and refusal responses, whose style affects results.
• Concept modules require per-category concept sets defined in advance, adding a fixed set of concepts per forget category.

Experimental setup

Datasets: safety QA benchmark (six types, ten categories each) ImageNet-R

Benchmarks: MMBench SEEDBench ScienceQA ImageNet-R

Metrics: Context-aware Refusal Rate Refusal Gap Answer Rate BERTScore CLIP Score ROUGE-L Specificity

Baselines: EWC LwF GMM EProj MoEAdapter SCRUB O^3

Builds on: MoEAdapter O^3 MiniGPT-4 Concept Bottleneck Models

Closest papers by meaning

• Don't Forget Why You Started: Tackling Dual Forgetting in Vision-Language Continual Learning (ICML 2026 · similarity 0.87)
• AUVIC: Adversarial Unlearning of Visual Concepts for Multi-modal Large Language Models (AAAI 2026 · similarity 0.87)
• Stabilizing Zero-Shot Prediction: A Novel Antidote to Forgetting in Continual Vision-Language Tasks (NeurIPS 2024 · similarity 0.86)
• Enhancing Visual Continual Learning with Language-Guided Supervision (CVPR 2024 · similarity 0.86)
• Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.86)
• In-Context Learning Can Help Vision Language Models Overcome Training Prior (NeurIPS 2026 · similarity 0.86)