Research map / Vision-language and multimodal / Human and embodied modeling
Quality over Quantity: Boosting Data Efficiency Through Ensembled Multimodal Data Curation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EcoDatum, a weak-supervision ensemble of unimodal and multimodal curation operators with quality-guided deduplication and search-based LF tuning, which ranks 1st on the DataComp small-scale filtering track.
Problem
Curating noisy, heterogeneous web-crawled image-text data so that CLIP-style models trained on a smaller subset perform better than on unfiltered or heuristically filtered data.
Approach
Combines heuristic unimodal filters (blur, geometry, language ID, ICC) and multimodal alignment scores (GroundingDINO box counts, CLIP, horizontally/vertically flipped CLIP) as labeling functions in a LabelModel weak-supervision ensemble, tuned by a composite metric (F1-tiny plus overlap/conflict/coverage) on a small clean-vs-noisy set, after quality-guided deduplication that keeps the best-aligne
Key results
- EcoDatum reaches average score 0.182 on 38 DataComp tasks using 3.5M of 12.8M pairs, versus 0.132 for no filtering and 0.142 for DataComp basic filtering (about 28% over baseline).
- Ranks 1st on the DataComp small-scale filtering leaderboard, ahead of T-MARS and WS (both 0.180).
- Ablation: quality-guided deduplication gives 1.4% over random dedup at equal size; combined unimodal and multimodal ensembles give 45.4% over no filtering.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace random deduplication with quality-guided deduplication (ablation) | DataComp curation · DataComp (38 downtasks) | Avg. Perf.: 0.145 -> 0.147 | Helped |
| add a unimodal operator ensemble after QGD (ablation) | DataComp curation · DataComp (38 downtasks) | Avg. Perf.: 0.147 -> 0.154 | Helped |
| add a multimodal operator ensemble after QGD (ablation) | DataComp curation · DataComp (38 downtasks) | Avg. Perf.: 0.147 -> 0.164 | Helped |
| combine unimodal and multimodal operator ensembles with QGD (ablation) | DataComp curation · DataComp (38 downtasks) | Avg. Perf.: 0.132 -> 0.182 | Helped |
| use global multimodal alignment rather than local multimodal alignment with unimodal operators and QGD (ablation) | DataComp curation · DataComp (38 downtasks) | Avg. Perf.: 0.155 -> 0.168 | Helped |
| use a single global alignment operator instead of no filtering (ablation) | DataComp curation · DataComp small-scale filtering | Avg. Perf.: 0.132 -> 0.160 | Helped |
| remove vertical-flip CLIP from global alignment (ablation) | EcoDatum · DataComp benchmark | Perf. | Hurt |
| replace optimized LabelModel weights with random weights (ablation) | EcoDatum operator ensemble · DataComp curation | Perf.: 0.182 vs. 0.157; 0.141; 0.161 | Hurt |
| increase the dataset size while using EcoDatum instead of no filtering (ablation) | EcoDatum · DataComp small-scale filtering | Avg. Perf.: 0.073 -> 0.075; 0.088 -> 0.104; 0.132 -> 0.182 | Helped |
Surprising findings
- Random initialization of LabelModel weights yields 0.141 to 0.161, with one configuration falling below the single-operator results, indicating the tuned composite-metric search matters.
- Single-operator results show global CLIP-based alignment (H-CLIP at 0.160 with 4.3M samples) outperforms the more elaborate local GroundingDINO operator (0.141).
Limitations
- Validated only on the small 12.8M-pair DataComp track; scalability to larger datasets is untested.
- Composite-metric coefficients are set by a few rounds of experiments rather than principled optimization.
- Improvement over the 0.180 WS and T-MARS entries is marginal (0.002).
Experimental setup
Datasets: DataComp small-scale filtering pool (12.8M Common Crawl image-text pairs) COCO (clean samples for tiny labeled set)
Benchmarks: DataComp 38-task evaluation suite (ImageNet, ImageNet distribution shifts, VTAB, retrieval)
Metrics: average performance over 38 tasks F1-tiny
Baselines: No Filtering LAION CLIP score filtering DataComp basic filtering HYPE LINE T-MARS WS (UW-Madison)
Builds on: DataComp Snorkel LabelModel GroundingDINO CLIP ICC
Closest papers by meaning
- The State of Data Curation at NeurIPS: An Assessment of Dataset Development Practices in the Datasets and Benchmarks Track (NeurIPS 2024 · similarity 0.86)
- Scaling Laws for Data Filtering-- Data Curation cannot be Compute Agnostic (CVPR 2024 · similarity 0.85)
- Data curation via joint example selection further accelerates multimodal learning (NeurIPS 2024 · similarity 0.85)
- SELECT: A Large-Scale Benchmark of Data Curation Strategies for Image Classification (NeurIPS 2024 · similarity 0.84)
- DataRater: Meta-Learned Dataset Curation (NeurIPS 2025 · similarity 0.84)
- Curated LLM: Synergy of LLMs and Data Curation for tabular augmentation in low-data regimes (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.