Research map / Vision-language and multimodal / Cross-modal learning
MultiMem: Measuring and Mitigating Memorization in Multi-Modal Contrastive Learning
Wenhao Wang, Franziska Boenisch, Michael Backes, Adam Dziedzic
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add noise augmentation to all three modalities instead of one (ablation) | AudioCLIP · AudioCLIP UrbanSound8K | AIT MultiMem / AI-T T@5: 0.332 -> 0.264 MultiMem; 36.9 -> 43.1 T@5 | Improved |
| apply noise to a single modality only (ablation) | AudioCLIP · AudioCLIP retrieval | T@5 on unaugmented pairs: I-T 33.1 -> 32.2 when audio augmented | Mixed |
| in-training noise on top 5% MultiMem-memorized samples | AudioCLIP · AudioCLIP UrbanSound8K retrieval | retrieval T@5: 36.9% -> 43.3% | Improved |
| post-training remove most memorized samples and fine-tune | AudioCLIP · AudioCLIP UrbanSound8K retrieval | retrieval T@5: 36.9% -> 39.3% | Improved |
| select samples by MultiMem rather than loss or gradient (ablation) | in-training noise mitigation · AudioCLIP retrieval | retrieval T@5: 39.9% (loss) -> 43.3% | Improved |
| add noise to random samples (ablation) | in-training mitigation · AudioCLIP retrieval | retrieval T@5: 36.9% -> 37.2% | No clear effect |
| add noise to least memorized samples (ablation) | in-training mitigation · AudioCLIP retrieval | retrieval T@5: 36.9% -> 36.1% | Hurt |
| modality-specific augmentations instead of Gaussian noise (ablation) | in-training mitigation · AudioCLIP retrieval | retrieval T@5: 43.3% -> 42.5% | No clear effect |
| ImageBind loss instead of pairwise alignment of all modalities (ablation) | AVIT-CLIP · MSR-VTT AVIT | MultiMem / retrieval: 0.488 -> 0.571 MultiMem; 48.2% -> 36.5% retrieval | Hurt |
| balance per-pair loss weights negatively correlated to memorization (ablation) | AudioCLIP · AudioCLIP | MultiMem: 0.332 -> 0.290 | Improved |
| vary candidate set size S_C from 500 to 2000 (ablation) | MultiMem metric · AVT-CLIP MSR-VTT | average MultiMem: 0.405 -> 0.408 | No clear effect |
Closest papers by meaning
- Quantifying Cross-Modality Memorization in Vision-Language Models (NeurIPS 2025 · similarity 0.91)
- Captured by Captions: On Memorization and its Mitigation in CLIP Models (ICLR 2025 · similarity 0.89)
- Mitigating Intra- and Inter-modal Forgetting in Continual Learning of Unified Multimodal Models (NeurIPS 2025 · similarity 0.87)
- On the Value of Cross-Modal Misalignment in Multimodal Representation Learning (NeurIPS 2025 · similarity 0.87)
- TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs (NeurIPS 2026 · similarity 0.87)
- Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
