Research map / Vision-language and multimodal / CLIP and zero-shot
Sieve: Multimodal Dataset Pruning using Image Captioning Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Sieve, a multimodal data pruning signal that scores image-text alignment by comparing alt-text to nucleus-sampled captions from a BLIP captioner using a sentence transformer, and fuses it with CLIPScore for DataComp pruning.
Problem
CLIPScore-based pruning of noisy web-crawled image-text data suffers from false positives (misaligned pairs scored high) and false negatives (aligned hard pairs scored low), hurting CLIP pretraining quality.
Approach
Use captions generated by an image-captioning model pretrained on small, curated data (BLIP, 14M pairs) with nucleus sampling, mask medium phrases like "image of", and measure alt-text alignment via cosine similarity in the embedding space of a lightweight sentence transformer (all-MiniLM-L6-v2); fuse with CLIPScore.
Key results
- On DataComp medium scale, Sieve+CLIPScore improves the 38-task average over CLIPScore by 2.6% (35.4 vs 32.8) and retrieval by 2.7%.
- On DataComp large scale, Sieve+CLIPScore improves the 38-task average over CLIPScore by 1.7% (54.6 vs 52.9) and retrieval by 4.5%.
- Captioner pretrained on curated 14M pairs yields better alignment than the 129M noisy variant (34.03 vs 33.05 avg at 20% kept).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fuse Sieve alignment scores with CLIPScore | Sieve on DataComp medium · Average over 38 datasets | average score: 32.8 -> 35.4 | Helped |
| Use Sieve pruning instead of CLIPScore pruning | CLIP pretraining on DataComp medium · Retrieval | score: 25.1 -> 28.9 | Helped |
| Fuse Sieve alignment scores with CLIPScore | Sieve on DataComp large · Average over 38 datasets | average score: 52.9 -> 54.6 | Helped |
| Use Sieve pruning instead of CLIPScore pruning | CLIP pretraining on DataComp large · Retrieval | score: 46.6 -> 52.0 | Helped |
| Use curated 14M BLIP pretraining data instead of BLIP-129M (ablation) | Sieve on DataComp medium · Average over 38 datasets | average score: 33.05 -> 34.03 | Helped |
| Use BLIP instead of GIT as the caption generator (ablation) | Sieve on DataComp medium · Average over 38 datasets | average score: 32.40 -> 34.03 | Helped |
| Replace the sentence transformer with the CLIP text encoder (ablation) | Sieve on DataComp medium · Average over 38 datasets | average score: 34.03 -> 25.90 | Hurt |
| Replace the sentence transformer with the BLIP text encoder (ablation) | Sieve on DataComp medium · Average over 38 datasets | average score: 34.03 -> 31.85 | Hurt |
| Increase generated captions per image from 1 to 8 (ablation) | Sieve on DataComp medium · Average over 38 datasets | average score: 32.50 -> 34.03 | Helped |
| Set CLIPScore fusion weight to 0.5 rather than 0.3 (ablation) | Sieve+CLIPScore on DataComp medium · Average over 38 datasets | average score: 34.40 -> 35.15 | Helped |
| Set CLIPScore fusion weight to 0.7 rather than 0.5 (ablation) | Sieve+CLIPScore on DataComp medium · Average over 38 datasets | average score: 35.15 -> 34.35 | Hurt |
Surprising findings
- Captioning models pretrained on a much larger noisy 129M dataset give worse pruning signals than the curated 14M set, despite higher captioning performance.
- Sieve alone underperforms CLIPScore on OCR tasks (MNIST, SVHN, Rendered SST-2) and Country211, though fusion recovers these.
Limitations
- Sieve underperforms CLIPScore on OCR-heavy tasks and context-based tasks like Country211.
- Gains are largest when fused with CLIPScore; Sieve alone is not uniformly better on average at large scale (52.3 vs 52.9).
- Relies on medium-phrase list and captioner-generated captions whose diversity is limited.
Experimental setup
Datasets: DataComp medium pool (128M) DataComp large pool (1.28B) BLIP pretraining data (14M curated; 129M variant) COCO Visual Genome Conceptual Captions Conceptual 12M SBU captions
Benchmarks: DataComp 38 downstream tasks ImageNet ImageNet distribution shifts VTAB retrieval tasks
Metrics: zero-shot accuracy retrieval accuracy average over 38 tasks
Baselines: No Filtering Basic Filtering LAION Filtering CLIPScore GIT-based Sieve CLIP text encoder BLIP text encoder
Builds on: CLIP DataComp BLIP CLIPScore all-MiniLM-L6-v2
Closest papers by meaning
- Mostly Text, Smart Visuals: Asymmetric Text-Visual Pruning for Large Vision-Language Models (CVPR 2026 · similarity 0.89)
- Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training (AAAI 2024 · similarity 0.89)
- CapsFusion: Rethinking Image-Text Data at Scale (CVPR 2024 · similarity 0.89)
- A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions (CVPR 2024 · similarity 0.89)
- Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models (AAAI 2025 · similarity 0.88)
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.