Research map / Vision-language and multimodal / Human and embodied modeling

MemLeak: Diagnosing Information Leaks in Multimodal Agent Memory

NeurIPS 2026 · Vision-language and multimodal › Human and embodied modeling · approach: Multimodal retrieval-augmented modeling · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces the Information Provenance Graph taxonomy and the MemLeak benchmark to measure how deleted user facts remain recoverable from retained text and images in multimodal agent memory systems.

Problem

Fact-level forgetting in multimodal agent memory often deletes only text entries, leaving the deleted fact recoverable from retained correlated text and user images via VLM inference.

Approach

A provenance-graph taxonomy classifying each representation of a fact by deletion affordance, plus a deletion-cascade benchmark that separates text-correlation leakage, retained-image leakage, and same-fact leakage, with a content-aware semantic deletion mitigation.

Key results

• Retained-image recovery leaks 12.0% (36/300) of deleted facts with a 0.0% blind baseline and 0.3% negative-control FPR, versus 48.7% same-fact recovery.
• Retained-text-only inference yields 18.3% recovery; 47% of image-leaked facts are not recovered from text alone.
• Content-aware semantic deletion reduces the retained-image residual from 12.0% to 2.0% while flagging 21.0% of retained images.

Surprising findings

• Images tagged to entirely unrelated facts still allow recovery of deleted facts at 12.0%.
• Direct probing of deletion-capable text-only systems yields under 1% leakage, but behavioral suppression leaks 17%.

Limitations

• Profiles and images are synthetic; real Unsplash photos are stock photos rather than in-situ user data.
• Only one deletion policy (text plus tagged images) is evaluated at scale; Letta is a simulated wrapper.
• Judge ensemble agreement with humans is lower on split votes (37.5%) and same-fact cases (kappa 0.47).

Experimental setup

Datasets: MemLeak synthetic profiles Unsplash real photographs

Benchmarks: MemLeak

Metrics: cross-modal leakage rate (CMLR) recovery rate with 95% Wilson CI false positive rate on negative controls Cohen's kappa

Baselines: Long-context Naive text deletion Mem0 Letta (simulated) Oracle provenance cascade GPT-4o retained-image probing

Builds on: FIDES MemoryAgentBench CoALA Mem0

Closest papers by meaning

• Personal Visual Memory from Explicit and Implicit Evidence (NeurIPS 2026 · similarity 0.89)
• SMMBench: A Benchmark for Source-Distributed Multimodal Agent Memory (NeurIPS 2026 · similarity 0.88)
• WorldMemArena: Evaluating Multimodal Agent Memory Through Action–World Interaction (NeurIPS 2026 · similarity 0.86)
• XFeat Revisited: Reproducibility and Evaluation of a Lightweight Image Matcher (NeurIPS 2026 · similarity 0.86)
• [Re] Boosting the Visual Interpretability of CLIP via Adversarial Fine-Tuning (NeurIPS 2026 · similarity 0.86)
• ViLoMem: Agentic Learner with Grow-and-Refine Multimodal Semantic Memory (CVPR 2026 · similarity 0.85)