Research map / Vision-language and multimodal / Human and embodied modeling
Multi-Modal Representation Learning With Text-Driven Soft Masks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add text-driven SoftMask (ablation) | SoftMask++ pretraining · MS-COCO fine-tuned image-text retrieval | TR@1: 73.10 -> 75.74 | Helped |
| add focal ITC loss (ablation) | SoftMask++ pretraining · MS-COCO fine-tuned image-text retrieval | TR@1: 73.10 -> 75.66 | Helped |
| add multimodal data augmentation (ablation) | SoftMask++ pretraining · MS-COCO fine-tuned image-text retrieval | TR@1: 73.10 -> 75.26 | Helped |
| replace text-driven soft masks with random hard masks (p=0.3) (ablation) | SoftMask++ · MS-COCO fine-tuned retrieval | TR@1: 76.62 -> 75.82 | Hurt |
| replace text-driven soft masks with random hard masks (p=0.5) (ablation) | SoftMask++ · MS-COCO fine-tuned retrieval | TR@1: 76.62 -> 76.22 | Hurt |
| also apply SoftMask to MLM (ablation) | SoftMask++ with SoftMask on ITM · MS-COCO fine-tuned retrieval | TR@1: 76.62 -> 76.21 | Hurt |
| use normalized cross-attention instead of Grad-CAM to generate masks (ablation) | SoftMask++ · MS-COCO fine-tuned retrieval | TR@1: 76.62 -> 75.68 | Hurt |
| combine SoftMask, focal ITC, and MMDA (ablation) | SoftMask++ pretraining · MS-COCO fine-tuned retrieval | TR@1: 73.10 -> 76.62 | Helped |
| pretrain with SoftMask++ rather than TCL | vision-language representation learning · Flickr30K fine-tuned text retrieval | R@1: 94.9 -> 95.4 | Helped |
| pretrain with SoftMask++ rather than TCL | vision-language representation learning · MS-COCO zero-shot image retrieval | R@1: 53.5 -> 54.1 | Helped |
Closest papers by meaning
- Linguistics-aware Masked Image Modeling for Self-supervised Scene Text Recognition (CVPR 2025 · similarity 0.88)
- MMRL: Multi-Modal Representation Learning for Vision-Language Models (CVPR 2025 · similarity 0.87)
- ITO: Multi-View Alignment and Training-Time Fusion for Image-Text Pretraining (NeurIPS 2026 · similarity 0.87)
- SyncMask: Synchronized Attentional Masking for Fashion-centric Vision-Language Pretraining (CVPR 2024 · similarity 0.87)
- T-MARS: Improving Visual Representations by Circumventing Text Feature Learning (ICLR 2024 · similarity 0.86)
- Improving fine-grained understanding in image-text pre-training (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.