Research map / Vision-language and multimodal / CLIP and zero-shot
CLIPLoss and Norm-Based Data Selection Methods for Multimodal Contrastive Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes s-CLIPLoss (CLIPScore plus a batch-based normalization term from the CLIP loss) and NormSim (p-norm similarity of target-set image embeddings) as training-free, OpenAI-CLIP-based data selection metrics for DataComp-medium.
Problem
Data selection for CLIP pretraining on noisy web-curated image-text data: existing CLIP-embedding-based filtering relies on the classical CLIPScore, which is biased, and the best methods require external models or newly trained CLIP-style teachers.
Approach
Replace CLIPScore with s-CLIPLoss, which subtracts an estimated normalization term from the CLIP contrastive loss computed over random batches, so that samples matching many contrastive pairs are penalized. Also introduce NormSim, a vision-only p-norm similarity between training images and target-task images, plus a dynamic proxy variant that needs no target data.
Key results
- s-CLIPLoss (30%) ∩ NormSim∞(Target) with OAI CLIP-L/14 reaches 31.7-31.9% on IN-1k and 35.0% average on 38 tasks, versus 26.4% IN-1k for the best CLIPScore baseline (5.3% gain on IN-1k, 2.8% on average).
- Combining with DFN ∪ HYPE reaches 37.7% average on 38 tasks, a 0.9% gain over the previous SOTA DFN ∪ HYPE (36.8%) on DataComp-medium.
- s-CLIPLoss improves over CLIPScore across teacher models, with gains of roughly 0.4% to 5.4% on IN-1k depending on model and ratio.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace CLIPScore with s-CLIPLoss | DataComp-medium selection with OAI CLIP-L/14 · DataComp-medium (110M downloaded) | Avg. (38 tasks): 32.2 -> 32.9 | Helped |
| Add NormSim∞(Target) to s-CLIPLoss selection | DataComp-medium selection with OAI CLIP-L/14 · DataComp-medium (110M downloaded) | Avg. (38 tasks): 32.9 -> 35.0 | Helped |
| Use NormSim∞(Target) rather than NormSim∞(IN-1k) with s-CLIPLoss (ablation) | s-CLIPLoss (30%), OAI CLIP-L/14 · DataComp-medium (110M downloaded) | Avg. (38 tasks): 34.4 -> 35.0 | Helped |
| Add proxy-target NormSim2-D to s-CLIPLoss | DataComp-medium selection with OAI CLIP-L/14 · DataComp-medium (110M downloaded) | Avg. (38 tasks): 32.9 -> 34.1 | Helped |
| Combine s-CLIPLoss with NormSim∞(Target) | DataComp-medium selection with OAI CLIP-B/32 · DataComp-medium (110M downloaded) | Avg. (38 tasks): 33.6 -> 35.2 | Helped |
| Use NormSim∞(Target) with s-CLIPLoss (ablation) | DataComp-medium selection with DFN-P · DataComp-medium (110M downloaded) | Avg. (38 tasks): 34.7 -> 34.4 | Hurt |
| Calculate NormSim∞ with OAI CLIP-B/32 instead of DFN-P embeddings (ablation) | DFN-P s-CLIPLoss selection · DataComp-medium (110M downloaded) | Avg. (38 tasks): 34.7 -> 35.7 | Helped |
| Add Ours (10%) to DFN ∪ HYPE | Full DataComp-medium benchmark · DataComp-medium (full 128M dataset) | Avg.: 37.9 -> 38.8 | Helped |
| Add Ours (10%) to DFN ∪ HYPE | Full DataComp-medium benchmark · DataComp-medium (full 128M dataset) | ImageNet-1k: 38.2 -> 38.2 | No clear effect |
| Merge Ours (10%)* with DFN and HYPE | DFN ∪ HYPE selection on downloaded DataComp-medium · DataComp-medium (110M downloaded) | Avg. (38 tasks): 36.8 -> 37.7 | Helped |
| Use s-CLIPLoss instead of CLIPScore | DFN-P teacher embeddings at 17.5% selection · DataComp-medium | Avg. (38 tasks): 33.8 -> 34.7 | Helped |
Surprising findings
- NormSim using DFN-P embeddings for target data hurts performance, because DFN-P is optimized for filtering at the expense of downstream accuracy; using OAI CLIP-B/32 for the NormSim computation instead recovers gains.
- Choosing data by s-CLIPLoss (lower loss) rather than higher CLIPScore works better, and a training-data-only proxy target (NormSim-D) gives a 1.9% average gain without any target data.
Limitations
- Experiments use only DataComp-medium (128M pool, about 110M downloaded); larger pools (large, xlarge) are not evaluated.
- NormSim relies on the quality of the teacher's embeddings on target data, so weak filtering-oriented teachers such as DFN-P degrade it.
- Target-based NormSim requires access to downstream training data, and the proxy variant depends on the training set itself.
Experimental setup
Datasets: DataComp-medium ImageNet-1k DFN public dataset (CC12M, CC3M, SS15M)
Benchmarks: DataComp-medium 38-task evaluation ImageNet-1k ImageNet distribution shifts VTAB (11 tasks) WILDS (3 tasks) Flickr30k MSCOCO WinoGAViL
Metrics: zero-shot accuracy average accuracy over 38 tasks
Baselines: CLIPScore Image-based filtering D2 Pruning T-MARS Devil MLM HYPE DFN DFN ∪ HYPE
Builds on: CLIP DataComp CLIPScore DFN HYPE
Closest papers by meaning
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.90)
- Meta CLIP 2: A Worldwide Scaling Recipe (NeurIPS 2025 · similarity 0.90)
- Demystifying CLIP Data (ICLR 2024 · similarity 0.89)
- FG-CLIP: Fine-Grained Visual and Textual Alignment (ICML 2025 · similarity 0.89)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.89)
- CLIP the Bias: How Useful is Balancing Data in Multimodal Learning? (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.