Research map / Learning theory and algorithms / Prediction and decision losses
Data curation via joint example selection further accelerates multimodal learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes JEST, which selects jointly learnable sub-batches from large super-batches using contrastive losses and reference-model scores, plus Flexi-JEST using multi-resolution FlexiViT scoring to cut compute overhead.
Problem
Data curation for large-scale multimodal contrastive pretraining typically scores examples independently, ignoring how examples in a batch interact, and online scoring of large super-batches adds substantial compute overhead.
Approach
Contrastive loss decomposes into conditional terms, so batch-level learnability can be sampled sequentially in chunks (blocked-Gibbs-like) from a super-batch, and the scorer can be a low-resolution approximation of the learner trained jointly at multiple resolutions to cut scoring FLOPs.
Key results
- JEST++ (ViT-B/16) reaches SigLIP 40B-level average performance with 4B examples, using 10x fewer iterations and ~4x less compute (23% total FLOPs vs. SigLIP).
- Flexi-JEST++ uses 11% of SigLIP's total FLOPs while achieving +0.9 mean delta over SigLIP 40B.
- On LAION-2B, JEST++ reaches 66.8 IN1K zero-shot and 54.8 COCO vs. 64.1/48.1 for LAION-440M using 12.8B examples, with 1.3B examples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use joint learnability batch selection instead of uniform sampling | JEST on WebLI · ImageNet zero-shot, ImageNet 10-shot, and COCO retrieval | average downstream performance: up to 6% | Helped |
| Select batches jointly rather than independently selecting examples | JEST with the same reference model · ImageNet and COCO downstream tasks | average downstream performance | Helped |
| Use easy-reference batch selection instead of independent example selection | Easy-reference JEST · ImageNet and COCO downstream tasks | average downstream performance | Helped |
| Increase filtering ratio to 90% | JEST learnability selection · ImageNet and COCO downstream tasks | training examples to reach the 3B-uniform baseline's final performance: 2B, 1B, and 0.67B | Helped |
| Use low-resolution scoring and multi-resolution training | Flexi-JEST · ImageNet and COCO downstream tasks | FLOPs at matched average performance: 9.9× fewer FLOPs | Helped |
| Use a strongly curated reference model rather than weak or moderate curation | JEST pretraining · ImageNet and COCO downstream tasks | performance improvement relative to uniform baseline: 2.7% improvement | Helped |
| Scale the curated reference dataset to WebLI-curated++ | JEST++ · ImageNet and COCO downstream tasks | average downstream performance | Helped |
| Remove offline image-text alignment filtering (ablation) | JEST++ on WebLI · ImageNet-1K zero-shot | accuracy: 76.9 vs. 76.7 | No clear effect |
| Remove offline image-text alignment filtering (ablation) | JEST++ on WebLI · COCO retrieval | average retrieval: 61.8 vs. 61.8 | No clear effect |
| Replace sigmoid contrastive loss with softmax contrastive loss (ablation) | JEST · ImageNet and COCO downstream tasks | relative improvement over uniform sampling | No clear effect |
| Use hard-learner scoring instead of learnability scoring (ablation) | Joint batch selection · ImageNet and COCO downstream tasks at higher filtering ratios | performance | Hurt |
| Train the approximate low-resolution model on too little data (ablation) | Flexi-JEST · ImageNet and COCO downstream tasks | performance relative to IID | Hurt |
Surprising findings
- Reference model performance is decoupled from (or anti-correlated with) JEST performance across curation strengths, with more aggressive curation giving weaker standalone models but better JEST gains.
- JEST performance is nearly identical when pretraining on unfiltered versus filtered WebLI, removing the need for pre-filtering.
Limitations
- Relies on small, well-curated reference datasets that specify the target distribution.
- Evaluation focuses on canonical downstream multimodal transfer tasks (ImageNet, COCO, fine-grained classification).
- Re-weighting selected data reduced performance, and the importance-sampling interpretation is left for future work.
Experimental setup
Datasets: WebLI WebLI-curated WebLI-curated++ LAION-2B
Benchmarks: ImageNet-1K 10-shot ImageNet-1K zero-shot COCO image-to-text retrieval COCO text-to-image retrieval Birds Caltech Cars Pets
Metrics: top-1 accuracy recall mean accuracy FLOPs relative to SigLIP
Baselines: SigLIP CLIP EVA-CLIP OpenCLIP LessIsMore SILC SemDeDup DBP LAION-440M
Builds on: SigLIP FlexiViT Prioritized sampling Reducible holdout loss selection
Closest papers by meaning
- Active Data Curation Effectively Distills Large-Scale Multimodal Models (CVPR 2025 · similarity 0.89)
- Continual Multimodal Contrastive Learning (NeurIPS 2025 · similarity 0.88)
- Breaking the Batch Barrier (B3) of Contrastive Learning via Smart Batch Mining (NeurIPS 2025 · similarity 0.87)
- When Dynamic Data Selection Meets Data Augmentation: Achieving Enhanced Training Acceleration (ICML 2025 · similarity 0.87)
- A CLIP-Powered Framework for Robust and Generalizable Data Selection (ICLR 2025 · similarity 0.87)
- Understanding the Gain from Data Filtering in Multimodal Contrastive Learning (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.