Research map / Vision-language and multimodal / Multimodal LLMs
Knowledge Externalization: Reversible Unlearning and Modular Retrieval in Multimodal Large Language Models
Jiaqi Li, Zihan You, Ruoyan Shen, Shenyu Zhang, Songlin Zhai, Yongrui Chen, Chuanyi Zhang, Jiahui Geng, Fakhri Karray, Sheng Bi, Guilin Qi
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes knowledge externalization, which moves sensitive knowledge out of MLLM parameters into a modular, reversible store so it can be unlearned and retrieved on demand.
Problem
Machine unlearning in multimodal LLMs irreversibly alters parameters to erase sensitive personal or proprietary knowledge, which conflicts with privacy regulations requiring auditable and reversible handling of data.
Approach
Instead of permanently erasing knowledge from weights, externalize it into a separate modular component that can be removed from the model (unlearning) and later reattached via retrieval (reversibility), so the base parameters stay intact.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Soft Orthogonal Weighting to DSM for triple-concept externalization (ablation) | DSM · MEXBench Trump&Chihuahua&Musk | GEN: 34.0 -> 97.0 | Improved |
| add Soft Orthogonal Weighting to DSM for triple-concept externalization (ablation) | DSM · MEXBench Trump&Hello Kitty&Harry Potter | GEN: 64.7 -> 92.7 | Improved |
| add Soft Orthogonal Weighting to DSM for dual-concept externalization (ablation) | DSM · MEXBench Trump&Chihuahua REC2 (InternVL3 2B) | REC2: 24 -> 90 | Improved |
| train memory token and model simultaneously (DSM) rather than sequential forget-then-recover (SFR) | externalization training · MEXBench single Trump (LLaVA 7B) | REC: 6 -> 100 | Improved |
| update both streams in the same step (DSM) rather than alternating tuning (AT) | externalization training · MEXBench single Elon Musk (LLaVA 7B) | GEN: 51 -> 91 | Improved |
| add SOW with lambda=0 (no regularization) versus lambda=0.5 (ablation) | DSM w/ SOW · Trump&Chihuahua | AVG_REC: 36.5 -> 99.5 | Improved |
| increase memory token length from 32 to 100-128 (single concept) (ablation) | DSM · Trump single concept | GEN: 4 -> 100 | Improved |
| increase memory token length to 256 (single concept) (ablation) | DSM · Trump single concept | GEN: 100 -> 55 | Hurt |
| increase number of externalized concepts from 2 to 8 (fixed training steps) (ablation) | DSM w/ SOW · MEXBench multi-concept | GEN and AVG_REC: GEN 97.0 -> 85.3 (2 vs 8 concepts) | Hurt |
| reverse the order of concatenated memory tokens (ablation) | token compositing · Trump&Chihuahua | REC1: 100 -> 5 | Hurt |
| externalize with SOW versus gradient-ascent-only SFR on general benchmark | InternVL-7B · MMBench (Trump&Chihuahua) | accuracy: 66.2 -> 76.2 | Improved |
| edit externalized memory token to a new target (Chihuahua to Samoyed vs Trump to Biden) | memory token editing · Knowledge editing | edit success: 100 (Biden) vs 75 (Samoyed) | Mixed |
Closest papers by meaning
- Towards Reasoning-Preserving Unlearning in Multimodal Large Language Models (CVPR 2026 · similarity 0.90)
- Large Scale Knowledge Washing (ICLR 2025 · similarity 0.89)
- A Closer Look at Machine Unlearning for Large Language Models (ICLR 2025 · similarity 0.88)
- Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference (NeurIPS 2024 · similarity 0.88)
- MLUBench: A Benchmark for Lifelong Unlearning Evaluation in MLLMs (ICML 2026 · similarity 0.88)
- MUSE: Machine Unlearning Six-Way Evaluation for Language Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
