Research map / Vision-language and multimodal / Cross-modal learning
MemEIC: A Step Toward Continual and Compositional Knowledge Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MemEIC, a hybrid external-internal editor with dual-modality retrieval memory, dual LoRA adapters and a LoRA-augmented knowledge connector, plus the CCKEB benchmark for continual compositional knowledge editing in LVLMs.
Problem
Knowledge editing for large vision-language models usually edits one modality in isolation and ignores continual sequential edits and compositional queries that combine visual and textual edits.
Approach
Separates visual and textual knowledge into modality-specific external retrieval memories and dual LoRA adapters, then uses a knowledge connector (LoRA in self-attention) that fuses the two adapters only when a query needs both modalities.
Key results
- Averaged across edit-test gaps on LLaVA-1.5, MemEIC reaches 98.93 visual reliability and 92.48 textual reliability.
- MemEIC averages 80.56 compositional reliability on LLaVA-1.5, +18.51 over the best baseline LoRA (62.05).
- With the Knowledge Connector, Dual-LoRA+RAG compositional reliability rises to 99.21% at gap 0 and 97.01% at gap 100.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine visual cues with textual cues for external-memory retrieval (ablation) | Mem-E · VLKEB sequential editing average, LLaVA-1.5 | Rel: 48.02 to 96.51 | Helped |
| combine visual cues with textual cues for external-memory retrieval (ablation) | Mem-E · VLKEB sequential editing average, LLaVA-1.5 | T-Gen: 50.74 to 93.42 | Helped |
| combine visual cues with textual cues for external-memory retrieval (ablation) | Mem-E · VLKEB sequential editing average, LLaVA-1.5 | I-Loc: 4.02 to 57.10 | Helped |
| separate visual and textual knowledge into Dual-LoRA adapters instead of one shared adapter (ablation) | Mem-I · CCKEB validation, visual edits | T-Loc: 63.16 to 80.93; +17.77% | Helped |
| separate visual and textual knowledge into Dual-LoRA adapters instead of one shared adapter (ablation) | Mem-I · CCKEB validation, textual edits | Rel: 74.54 to 77.24; +2.70% | Helped |
| reduce the shared LoRA rank from r:16 to r:8 (ablation) | Mem-I · CCKEB validation, textual edits | Rel: 74.54 to 72.12 | Hurt |
| add an attention-based Knowledge Connector to Dual-LoRA+RAG (ablation) | Dual-LoRA+RAG · CCKEB test set, compositional queries, gap=0 | compositional reliability: 78.16\,\% to 99.21\,\% | Helped |
| add an attention-based Knowledge Connector to Dual-LoRA+RAG (ablation) | Dual-LoRA+RAG · CCKEB test set, compositional queries, gap=100 | compositional reliability: 63.39\,\% to 97.01\,\% | Helped |
| use modality-separated adapters instead of a shared Single-LoRA adapter (ablation) | LoRA · CCKEB test set, compositional queries, gap=0 | compositional reliability: 77.47\% to 70.05\% | Hurt |
| add the Knowledge Connector to MemEIC | MemEIC · CCKEB test set, averaged across edit-test gaps, LLaVA-1.5 | compositional reliability: +18.51 points over the best baseline, LoRA (62.05) | Helped |
| use MemEIC instead of WISE | LLaVA-1.5 · CCKEB test set, averaged across edit-test gaps | visual reliability: +16.94 | Helped |
| use MemEIC instead of WISE | LLaVA-1.5 · CCKEB test set, averaged across edit-test gaps | compositional reliability: +32.35 | Helped |
Surprising findings
- Oracle retrieval alone (Base+RAG) plateaus at 64.93% compositional reliability, showing perfect retrieval is not sufficient without integration.
- Dual-LoRA without the connector underperforms Single-LoRA on compositional reliability at gap 0 (70.05% vs 77.47%).
Limitations
- Experiments limited to paired visual-then-textual edits and two LVLM backbones (LLaVA-1.5, MiniGPT-4).
- Query decomposition relies on GPT-4o, adding inference cost if run on-the-fly.
- Compositional evaluation largely uses an oracle external memory; realistic retrieval conflicts are only partially studied.
Experimental setup
Datasets: CCKEB VLKEB MMKG DB15K FB15K
Benchmarks: CCKEB VLKEB
Metrics: visual reliability textual reliability compositional reliability (CompRel) generality locality portability
Baselines: SERAC FT (LLM) LoRA MEND WISE
Builds on: SERAC WISE VLKEB MEND LoRA
Closest papers by meaning
- Towards Unified Multimodal Editing with Enhanced Knowledge Collaboration (NeurIPS 2024 · similarity 0.90)
- Lifelong Knowledge Editing for Vision Language Models with Low-Rank Mixture-of-Experts (CVPR 2025 · similarity 0.90)
- MMKE-Bench: A Multimodal Editing Benchmark for Diverse Visual Knowledge (ICLR 2025 · similarity 0.89)
- ComprehendEdit: A Comprehensive Dataset and Evaluation Framework for Multimodal Knowledge Editing (AAAI 2025 · similarity 0.89)
- VLKEB: A Large Vision-Language Model Knowledge Editing Benchmark (NeurIPS 2024 · similarity 0.89)
- Attribution Analysis Meets Model Editing: Advancing Knowledge Correction in Vision Language Models with VisEdit (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.