Research map / Vision-language and multimodal / Cross-modal learning
TiMix: Text-Aware Image Mixing for Effective Vision-Language Pre-training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TiMix, a text-aware CutMix-style mixing augmentation for vision-language contrastive pre-training that uses a learned text-aware patch predictor, and claims mutual-information justification and improved data efficiency for ALBEF and mPLUG.
Problem
Web-harvested image-text pairs are noisy and partially aligned, so SMCL-based VLP is data-inefficient and scaling data is computationally costly.
Approach
Mix images with CutMix-style region replacement where the source region has the highest text-relevance score and the target region the lowest, using a Text-aware Patch Predictor trained with a patch-text alignment task on object/region annotations; mixed samples feed contrastive losses with soft labels, and the authors argue they act as an implicit MI regularizer.
Key results
- mPLUG-TiMix with 14M pre-training data reaches 81.53 VQA std (test-std) versus 79.22 for mPLUG at 14M, per the paper's Table 1.
- ALBEF-TiMix at 4M improves VQA test-dev from 74.54 to 75.92 over ALBEF at 4M.
- mPLUG-TiMix with 4M data reports 40% of the data at 43.8% of training time matching the performance claims against mPLUG at 14M.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add TiMix to mPLUG pre-training | mPLUG · VQA2.0 dev (14M) | accuracy: 79.65 -> 80.83 | Helped |
| Add TiMix to mPLUG pre-training | mPLUG · NLVR2 Test-P (14M) | accuracy: 84.21 -> 85.22 | Helped |
| Add TiMix to mPLUG pre-training | mPLUG · COCO retrieval (5K test), TR R@1 (14M) | R@1: 80.6 -> 82.3 | Helped |
| Add TiMix to mPLUG pre-training | mPLUG · COCO retrieval (5K test), IR R@1 (14M) | R@1: 63.9 -> 65.2 | Helped |
| Add TiMix to mPLUG pre-training | mPLUG · Flickr30K retrieval (1K test), TR R@1 (14M) | R@1: 96.5 -> 97.2 | Helped |
| Add TiMix to mPLUG pre-training | mPLUG · Flickr30K retrieval (1K test), IR R@1 (14M) | R@1: 86.3 -> 87.9 | Helped |
| Add TiMix to mPLUG pre-training | mPLUG · COCO Caption Karpathy test, cross-entropy optimization | CIDEr: 136.8 -> 138.3 | Helped |
| Add TiMix to mPLUG pre-training | mPLUG · RefCOCO+ val (4M) | accuracy (IoU 0.5): 80.07 -> 82.11 | Helped |
| Add PTA to text-guided image mixing (ablation) | mPLUG-TiMix · VQA dev (4M) | accuracy: 77.89 -> 78.63 | Helped |
| Add PTA to text-guided image mixing (ablation) | mPLUG-TiMix · NLVR2 dev (4M) | accuracy: 82.59 -> 84.12 | Helped |
| Add mix-based augmentation while retaining PTA (ablation) | mPLUG-TiMix · VQA dev (4M) | accuracy: 78.21 -> 78.63 | Helped |
| Add mix-based augmentation while retaining PTA (ablation) | mPLUG-TiMix · NLVR2 dev (4M) | accuracy: 83.13 -> 84.12 | Helped |
Surprising findings
- Standard CutMix and Mixup give only limited gains, and random-region mixing without text guidance (w/o PTA) yields negligible improvement over the baseline.
Limitations
- Text-aware patch predictor relies on COCO/VG object and region annotations, which are not available for general web data.
- Gains from the PTA-only ablation are modest, and the MI analysis is a lower-bound argument under assumptions of clean data.
Experimental setup
Datasets: MS COCO Visual Genome Conceptual Captions Conceptual 12M SBU Captions VQA2.0 NLVR2 Flickr30K COCO Caption RefCOCO+
Benchmarks: VQA2.0 NLVR2 Flickr30K COCO retrieval COCO Caption RefCOCO+
Metrics: VQA accuracy accuracy recall@k BLEU@4 METEOR CIDEr SPICE IoU 0.5 accuracy
Baselines: ALBEF mPLUG BLIP METER VLMo SCL MAP SimVLM
Builds on: ALBEF mPLUG CutMix InfoNCE
Closest papers by meaning
- MergeMix: A Unified Augmentation Paradigm for Visual and Multi-Modal Understanding (ICLR 2026 · similarity 0.87)
- COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training (CVPR 2025 · similarity 0.87)
- Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment (NeurIPS 2024 · similarity 0.87)
- Text-Printed Image: Bridging the Image-Text Modality Gap for Text-centric Training of Large Vision-Language Models (CVPR 2026 · similarity 0.87)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.87)
- Scaling Laws for Multimodal Data Mixtures (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.