Research map / Vision-language and multimodal / Multimodal LLMs
Words Before Pixels: Selective Modality Routing for Vision-Language Model Unlearning
Laura Yao, Haochen Zhang, Jinhao Duan, Sijia Liu, Tianlong Chen
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes selective modality routing for VLM unlearning, deciding per forget example how to represent it (text-only vs. image-conditioned) before applying an unlearning objective.
Problem
Machine unlearning in vision-language models must remove undesirable behavior from multimodal forget data while preserving retained capabilities; existing methods keep image-conditioned forget representations fixed without considering how each example should be represented.
Approach
A data-centric view: rather than only changing the unlearning objective, selectively route each forget example's modality (words before pixels) prior to optimization, so that some forget examples are represented without image conditioning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Include text renderings in the forget set (text-heavy mixtures) rather than only multimodal (ablation) | NPO-style VLM unlearning · VLGuard, LLaVA-1.5-7B | Unsafe R/I: 0.353 -> 0.975 | Improved |
| Use text-only forget data instead of multimodal at matched 341 examples (ablation) | NPO-style unlearning · VLGuard, LLaVA-1.5-7B | Unsafe R/I: 0.326 -> 0.476 | Improved |
| Use text-only data for all examples (gamma=1) (ablation) | ModRoute · VLGuard, LLaVA-1.5-7B | U-U score: 0.847 -> 0.575 | Hurt |
| Switch half of examples to text (gamma=0.5) instead of none (ablation) | ModRoute · VLGuard, LLaVA-1.5-7B | U-U score: 0.527 -> 0.847 | Improved |
| Select switched examples by high image-gradient saliency instead of random (ablation) | ModRoute · VLGuard, LLaVA-1.5-7B, gamma=0.5 | U-U score: 0.581 -> 0.847 | Improved |
| Route low-gradient examples instead of high-gradient (ablation) | ModRoute · VLGuard, LLaVA-1.5-7B, gamma=0.5 | U-U score: 0.847 -> 0.664 | Hurt |
| Rank by image-gradient L2 norm instead of proposed saliency (ablation) | ModRoute · VLGuard, LLaVA-1.5-7B, gamma=0.5 | U-U score: 0.847 -> 0.715 (high-L2), 0.638 (low-L2) | Hurt |
| Gradient-guided instead of random routing at gamma=0.75 (ablation) | ModRoute · VLGuard and COVID memes | U-U score | No clear effect |
| Gradient-guided instead of random routing at gamma=0.25 (ablation) | ModRoute · COVID memes, LLaVA-1.5-7B | U-U score: 0.489 -> 0.731 (vs random) | Improved |
| Text-heavy mixture at matched 441 budget (ablation) | NPO-style unlearning · VLGuard, LLaVA-1.5-7B | Safe R/I: 0.170 (MM-heavy) -> 0.315 (text-heavy) | Hurt |
| Apply gradient-guided modality routing on top of NPO instead of plain multimodal NPO | NPO · VLGuard, LLaVA-1.5-7B | U-U score: 0.524 -> 0.847 | Improved |
Closest papers by meaning
- LoMo: Local Modality Substitution for Deeper Vision-Language Fusion (NeurIPS 2026 · similarity 0.91)
- Split to Merge: Unifying Separated Modalities for Unsupervised Domain Adaptation (CVPR 2024 · similarity 0.90)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.89)
- Mostly Text, Smart Visuals: Asymmetric Text-Visual Pruning for Large Vision-Language Models (CVPR 2026 · similarity 0.89)
- Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Models (NeurIPS 2024 · similarity 0.89)
- Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment (AAAI 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
