Research map / Vision-language and multimodal / Cross-modal learning
DPL: Decoupled Prototype Learning for Enhancing Robustness of Vision-Language Transformers to Missing Modalities
Jueqing Lu, Yuanyuan Qi, Xiaohao Yang, Shuaicheng Niu, Fucai Ke, Shujie Zhou, Wei Tan, Jionghao Lin, Wray Buntine, Hamid Rezatofighi, Lan Du
This paper tried a common change: Add a contrastive loss
add prototype relational contrastive loss L_PRC to ArcFace · Hateful Memes (mixed missing 70%) · AUROC: 68.59 -> 69.31 Improved
What 1,317 accepted papers report for this change
- 1,253 improved
- 22 no clear effect
- 16 mixed
- 26 hurt
- Beyond Looking Up, Try Looking Around: Harmonizing Global Structure and Local Consistency in Optimal Transport for Short Text Clustering ICML 2026 · add contrastive learning · SearchSnippets · ACC: 56.65 -> 80.12Improved
- SleepLM: Natural-Language Intelligence for Human Sleep ICML 2026 · add contrastive objective to captioning (CoCa vs Cap) · sleep event IoU · IoU: 29.2 -> 27.4Hurt
- LIMSSR: LLM-Driven Sequence-to-Score Reasoning under Training-Time Incomplete Multimodal Observations ICML 2026 · add contrastive loss L_con · FS1000 average · Spearman correlation: 0.759 -> 0.789Improved
See this change in detailMatched by the wording of each paper's own result, read by AI from the full text. Papers report gains more often than failures, so this shows what gets published, not the chance a change works.
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace FC head with DPL | MaPLe · MM-IMDb (text 50% missing) | F1-Macro: 54.31 -> 56.36 | Improved |
| replace FC head with DPL | DCP · MM-IMDb (text 50% missing) | F1-Macro: 53.62 -> 56.94 | Improved |
| replace FC head with DPL | MAP · MM-IMDb (text 50% missing) | F1-Macro: 53.32 -> 56.03 | Improved |
| replace FC head with DPL | frozen CLIP, no prompt tuning · UPMC Food-101 (mixed missing 90%) | Accuracy: 79.44 -> 81.99 | Improved |
| replace FC head with DPL | DCP · Hateful Memes (mixed missing 70%) | AUROC: 64.96 -> 67.68 | Improved |
| replace FC head with DPL | frozen CLIP, no prompt tuning · Hateful Memes (image 100%/text 10%) | AUROC: 63.48 -> 63.60 (text-only 10%); 66.61 -> 70.76 (image 100%/text 10% row, 90% rate) | Mixed |
| replace FC head with DPL | MAP · Hateful Memes (image 100%/text 30%, ViLT) | AUROC: 62.24 -> 63.29 | Improved |
| replace FC head with DPL | MAP · Hateful Memes (image 30%/text 100%, ViLT) | AUROC: 54.02 -> 64.53 | Improved |
| replace FC head with DePT | MAP · UPMC Food-101 (mixed missing 70%) | Accuracy: 85.13 -> 84.02 | Hurt |
| replace FC head with DePT | MAP · UPMC Food-101 (mixed missing 90%) | Accuracy: 79.54 -> 79.97 | Improved |
| decompose prototypes into image and text components (ablation) | DPL, DCP · Hateful Memes (mixed missing 70%) | AUROC: 66.82 -> 68.59 | Improved |
| decompose prototypes into modality components (ablation) | DPL, DCP · Hateful Memes (mixed missing 90%) | AUROC: 66.36 -> 64.22 (ArcFace decomp. vs un-decomp.) | Mixed |
Closest papers by meaning
- Deep Correlated Prompting for Visual Recognition with Missing Modalities (NeurIPS 2024 · similarity 0.87)
- Dual Prototype Evolving for Test-Time Generalization of Vision-Language Models (NeurIPS 2024 · similarity 0.87)
- Machine Vision Therapy: Multimodal Large Language Models Can Enhance Visual Robustness via Denoising In-Context Learning (ICML 2024 · similarity 0.87)
- LoMo: Local Modality Substitution for Deeper Vision-Language Fusion (NeurIPS 2026 · similarity 0.87)
- Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Data (CVPR 2025 · similarity 0.87)
- TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
