Research map / Vision-language and multimodal / Multimodal LLMs

Concept-Aware Batch Sampling Improves Language-Image Pretraining

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces DataConcept (128M DataComp images with concept tags, boxes, and concept-aware recaptions) and CABS, an online concept-aware batch sampler (diversity- and frequency-maximization variants) that improves CLIP/SigLIP pretraining.

Problem

Existing vision-language data curation is offline and concept-agnostic, using model-based filters that induce biases and discarding data permanently. The paper asks how to modulate visual concept distributions during language-image pretraining in a task-adaptive way.

Approach

Annotate a 128M web image-text pool with grounded concepts (RAM++ tags refined by GroundingDINO with multi-resolution WBF) and perform online batch sampling from superbatches by top-k scoring: a greedy diversity gain that balances concept counts (CABS-DM) or a concept-count gain favoring multi-object samples (CABS-FM).

Key results

• CABS-DM with ViT-B-32 CLIP raises ImageNet zero-shot accuracy from 17.3 (IID) to 21.9 on alt-texts, and average classification from 33.0 to 35.5 with recaptions.
• CABS-FM improves average retrieval for ViT-B-32 CLIP with recaptions from 32.6 (IID) to 41.6, and for SigLIP from 47.0 to 51.6.
• Under CLIP-score filtering (top 30%), CABS-DM improves average classification for ViT-B-32 CLIP from 35.7 to 37.8.

Surprising findings

• GRIT-VLP and MAFA fail to beat IID sampling for ViT-B-32 CLIP on classification and retrieval.
• MetaCLIP-style offline concept-balanced curation underperforms IID on some benchmarks such as Obj and Scene for CLIP.

Limitations

• Concept annotation is costly, though amortizable across training runs.
• Runtime of CABS increases with the filter ratio f.
• Not tested on complex multimodal architectures or large-scale state-of-the-art training setups.

Experimental setup

Datasets: DataComp DataComp-12.8B XLarge subset ImageNet MSCOCO Flickr30k

Benchmarks: ImageNet ImageNet-shift variants Let-It-Wag! MSCOCO Flickr30k

Metrics: zero-shot accuracy recall@k

Baselines: IID sampling MetaCLIP GRIT-VLP MAFA

Builds on: CLIP SigLIP DataComp MetaCLIP GRIT-VLP MAFA

Closest papers by meaning

• Bridging the gap to real-world language-grounded visual concept learning (NeurIPS 2025 · similarity 0.89)
• ConceptBed: Evaluating Concept Learning Abilities of Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.88)
• Concept-Guided Prompt Learning for Generalization in Vision-Language Models (AAAI 2024 · similarity 0.88)
• A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models (CVPR 2024 · similarity 0.88)
• The Neglected Tails in Vision-Language Models (CVPR 2024 · similarity 0.88)
• Language Models as Black-Box Optimizers for Vision-Language Models (CVPR 2024 · similarity 0.88)