Research map / Label-efficient and robust learning / Recognition and long-tail
Differential-informed Sample Selection Accelerates Multimodal Contrastive Learning
Zihua Zhao, Feng Hong, Mengxi Chen, Pengyi Chen, Benyuan Liu, Jiangchao Yao, Ya Zhang, Yanfeng Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| select samples by differential of CLIPScore (DISSect-Warmup) instead of random online selection | CLIP-ResNet50 on CC3M · MS-COCO 5K zero-shot, 30% selection | IR@1: 19.36 -> 21.34 | Improved |
| select by differential instead of small-loss trick | CLIP-ResNet50 on CC3M · MS-COCO 5K zero-shot, 50% selection | IR@1: 20.05 -> 23.58 | Improved |
| use temporal ensembling instead of warm-up for the historical score (ablation) | DISSect · MS-COCO 5K zero-shot, 50% selection | IR@1: 23.58 -> 22.60 | No clear effect |
| use temporal ensembling instead of warm-up for the historical score (ablation) | DISSect · Flickr30K 1K zero-shot, 50% selection | IR@1: 39.40 -> 42.70 | Mixed |
| add DISSect selection to CLIP pretraining on YFCC15M | CLIP-ResNet50 · MS-COCO 5K zero-shot, 50% selection | R@Sum: 162.43 -> 167.79 | Improved |
| add DISSect selection to CLIP pretraining on CC12M | CLIP-ResNet50 · MS-COCO 5K zero-shot, 50% selection | R@Sum: 178.79 -> 187.61 | Improved |
| replace SCAN with DISSect | BLIP-Base · NLVR2 dev | accuracy: 72.19 -> 73.99 | Improved |
| replace SCAN with DISSect | CLIP-ResNet101 · MS-COCO 5K zero-shot, 50% selection | R@Sum: 143.52 -> 150.18 | Improved |
| use single-network DISSect instead of dual-network GSC | CLIP-ResNet50 on CC3M · MS-COCO 5K zero-shot | R@Sum: 144.72 -> 148.44 | Improved |
| use loss-based Big Loss selection instead of random online | CLIP-ResNet50 on CC3M · MS-COCO 5K zero-shot, 30% selection | IR@1: 19.36 -> 15.54 | Hurt |
| use oracle-free DISSect instead of oracle CLIPScore | CLIP-ResNet50 on CC3M · MS-COCO 5K zero-shot, 70% selection | IR@1: 23.74 -> 22.74 | Hurt |
| apply DISSect to reach full-data performance | CLIP-ResNet50 on CC3M · MS-COCO retrieval | training iterations: approximately 2.85x acceleration | Improved |
Closest papers by meaning
- Data curation via joint example selection further accelerates multimodal learning (NeurIPS 2024 · similarity 0.88)
- Continual Multimodal Contrastive Learning (NeurIPS 2025 · similarity 0.86)
- Understanding the Gain from Data Filtering in Multimodal Contrastive Learning (NeurIPS 2025 · similarity 0.86)
- $\mathbb{X}$-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphs (ICLR 2025 · similarity 0.86)
- CLIPLoss and Norm-Based Data Selection Methods for Multimodal Contrastive Learning (NeurIPS 2024 · similarity 0.85)
- A CLIP-Powered Framework for Robust and Generalizable Data Selection (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
