AI Paper Atlas

Research map / Vision-language and multimodal / Multimodal LLMs

Words Before Pixels: Selective Modality Routing for Vision-Language Model Unlearning

Laura Yao, Haochen Zhang, Jinhao Duan, Sijia Liu, Tianlong Chen

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes selective modality routing for VLM unlearning, deciding per forget example how to represent it (text-only vs. image-conditioned) before applying an unlearning objective.

Problem

Machine unlearning in vision-language models must remove undesirable behavior from multimodal forget data while preserving retained capabilities; existing methods keep image-conditioned forget representations fixed without considering how each example should be represented.

Approach

A data-centric view: rather than only changing the unlearning objective, selectively route each forget example's modality (words before pixels) prior to optimization, so that some forget examples are represented without image conditioning.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Include text renderings in the forget set (text-heavy mixtures) rather than only multimodal (ablation)NPO-style VLM unlearning · VLGuard, LLaVA-1.5-7BUnsafe R/I: 0.353 -> 0.975Improved
Use text-only forget data instead of multimodal at matched 341 examples (ablation)NPO-style unlearning · VLGuard, LLaVA-1.5-7BUnsafe R/I: 0.326 -> 0.476Improved
Use text-only data for all examples (gamma=1) (ablation)ModRoute · VLGuard, LLaVA-1.5-7BU-U score: 0.847 -> 0.575Hurt
Switch half of examples to text (gamma=0.5) instead of none (ablation)ModRoute · VLGuard, LLaVA-1.5-7BU-U score: 0.527 -> 0.847Improved
Select switched examples by high image-gradient saliency instead of random (ablation)ModRoute · VLGuard, LLaVA-1.5-7B, gamma=0.5U-U score: 0.581 -> 0.847Improved
Route low-gradient examples instead of high-gradient (ablation)ModRoute · VLGuard, LLaVA-1.5-7B, gamma=0.5U-U score: 0.847 -> 0.664Hurt
Rank by image-gradient L2 norm instead of proposed saliency (ablation)ModRoute · VLGuard, LLaVA-1.5-7B, gamma=0.5U-U score: 0.847 -> 0.715 (high-L2), 0.638 (low-L2)Hurt
Gradient-guided instead of random routing at gamma=0.75 (ablation)ModRoute · VLGuard and COVID memesU-U scoreNo clear effect
Gradient-guided instead of random routing at gamma=0.25 (ablation)ModRoute · COVID memes, LLaVA-1.5-7BU-U score: 0.489 -> 0.731 (vs random)Improved
Text-heavy mixture at matched 441 budget (ablation)NPO-style unlearning · VLGuard, LLaVA-1.5-7BSafe R/I: 0.170 (MM-heavy) -> 0.315 (text-heavy)Hurt
Apply gradient-guided modality routing on top of NPO instead of plain multimodal NPONPO · VLGuard, LLaVA-1.5-7BU-U score: 0.524 -> 0.847Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy