Research map / Label-efficient and robust learning / Recognition and long-tail
Breaking the Batch Barrier (B3) of Contrastive Learning via Smart Batch Mining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes B3, a batch mining method that uses a teacher embedding model's rankings and METIS graph clustering to build batches of mutual strong in-batch negatives, achieving state-of-the-art on MMEB with LoRA-trained Qwen2-VL/InternVL3 models.
Problem
Contrastive learning for embedding models depends heavily on batch size and negative quality, but explicitly mining hard negatives is computationally costly, especially for multimodal models with high-resolution images.
Approach
Use a teacher's ranks over the full training set to build a sparse similarity graph of mutually strong negatives, cluster it with METIS into fixed-size groups, and form each batch from random clusters so in-batch negatives are hard, instead of adding separately mined hard negatives to random batches.
Key results
- B3++ (Qwen2-7B) reaches 72.0 average on MMEB, +1.3 over the prior best at 7B scale.
- B3++ (Qwen2-2B) reaches 68.1 average on MMEB, +2.9 over the prior best at 2B scale.
- B3 (Qwen2-2B, batch size 1024) beats Random Batches by 2.5 points (67.87 vs 65.35) with the same compute.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use B3++ batch mining and hard negatives instead of the strongest listed baseline | MMEB 2B models · MMEB 36 tasks (2B) | average accuracy: +2.9 points | Helped |
| use B3++ instead of the strongest listed baseline | MMEB 7B models · MMEB 36 tasks (7B) | average accuracy: +1.3 points | Helped |
| replace random batches with B3 batch mining | Qwen2-2B contrastive training · MMEB 36 tasks (batch size 1024) | average accuracy: 65.35 -> 67.87 | Helped |
| add positive instruction prompts (ablation) | Qwen2-2B with random batches · MMEB 36 tasks (batch size 1024) | average accuracy: 62.61 -> 65.35 | Helped |
| add B3 batch mining rather than use random batches (ablation) | Qwen2-2B contrastive training · MMEB 36 tasks (batch size 1024) | average accuracy: 65.35 -> 67.87 | Helped |
| replace random batches with B3 and remove the five additional hard negatives (ablation) | Qwen2-2B contrastive training · MMEB 36 tasks (batch size 1024) | average accuracy: 67.14 -> 67.87 | Helped |
| increase cluster size from K=8 to K=32 (ablation) | B3 Qwen2-2B · MMEB 36 tasks (batch size 512) | average accuracy: 67.40 -> 67.58 | Helped |
| increase cluster size from K=32 to K=512 (ablation) | B3 Qwen2-2B · MMEB 36 tasks (batch size 512) | average accuracy: 67.58 -> 67.28 | Hurt |
| increase maximum image resolution from 700 to 1000 (ablation) | B3 Qwen2-2B · MMEB 36 tasks (batch size 1024) | average accuracy: 67.45 -> 67.87 | Helped |
| replace the VLM2Vec 2B teacher with CLIP 400M (ablation) | B3 2B · MMEB 36 tasks (batch size 512, K=32) | average accuracy: 67.58 -> 67.30 | No clear effect |
| replace the VLM2Vec 2B teacher with VLM2Vec 7B (ablation) | B3 2B · MMEB 36 tasks (batch size 512, K=32) | average accuracy: 67.58 -> 67.57 | No clear effect |
| increase p from 30 to 70 (ablation) | B3 on VQA datasets · VQA tasks (batch size 32) | average accuracy: 57.37 -> 58.21 | Helped |
Surprising findings
- Using a stronger teacher (VLM2Vec 7B) gives minimal gains over the 2B teacher, and a weak CLIP teacher still performs competitively.
- B3 without any explicit hard negatives matches or beats random batches augmented with 5 hard negatives at half the compute.
Limitations
- Requires extra time and compute to rank the entire training set with a teacher model.
- Batch size and K are constrained; very large K degrades performance, so K must be tuned empirically.
- Sensitivity to p and m at small batch sizes; false negatives from small p hurt performance.
Experimental setup
Datasets: MMEB training set NLI data (SimCSE)
Benchmarks: MMEB (36 tasks) Flickr30K COCO Urban1k STS benchmarks
Metrics: accuracy recall@K
Baselines: Random Batches LLaVE UniME VLM2Vec mmE5 MMRet CLIP GBM
Builds on: VLM2Vec SFR-Embedding NV-Retriever METIS GritLM
Closest papers by meaning
- On the Similarities of Embeddings in Contrastive Learning (ICML 2025 · similarity 0.90)
- Data curation via joint example selection further accelerates multimodal learning (NeurIPS 2024 · similarity 0.87)
- $\mathbb{X}$-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphs (ICLR 2025 · similarity 0.87)
- A Statistical Theory of Contrastive Learning via Approximate Sufficient Statistics (NeurIPS 2025 · similarity 0.86)
- Weakly-Supervised Contrastive Learning for Imprecise Class Labels (ICML 2025 · similarity 0.86)
- BECLR: Batch Enhanced Contrastive Few-Shot Learning (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.