Research map / Vision-language and multimodal / Human and embodied modeling
COSA: Concatenated Sample Pretrained Vision-Language Foundation Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes COSA, which pretrains a vision-language foundation model on online-concatenated image-text samples that act as pseudo long-form video-paragraph corpora, achieving strong video and image-text results using only image-text data.
Problem
Vision-language foundation models pretrained on image-text data model visual semantics but ignore temporal/event-level correlations, while video-text corpora are small and noisy.
Approach
Randomly group image-text samples within a batch and concatenate their images and captions in order to form pseudo long-form video-paragraph inputs on the fly, with concatenated versions of ITC, ITM, MLM and generative objectives plus vanilla ITC/ITM on single samples.
Key results
- COSA (1.2B, 415M data) achieves 49.2 on MSRVTT-QA, 60.0 on MSVD-QA, 79.5 on TGIF-QA, and 49.9 on ActivityNet-QA, new state of the art among the compared models.
- COSA-B (17M) reaches R@1 64.1 on DiDeMo and 59.3 on ActivityNet retrieval, outperforming VINDLU-B and HiTeA at similar scale.
- In ablations with a frozen CLIP-B backbone on CC3M, COSA raises DiDeMo retrieval R@1 from 49.2 to 54.8 over the single-sample baseline, while COSA-copy drops it to 43.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply cross-sample concatenation rather than single-sample training (ablation) | COSA · DiDeMo | R@1: 49.2 -> 54.8 | Helped |
| Apply cross-sample concatenation rather than single-sample training (ablation) | COSA · VIST | CIDEr: 14.8 -> 20.7 | Helped |
| Add cross-sample concatenation instead of duplicating one sample (ablation) | COSA · DiDeMo | R@1: 43.5 -> 54.8 | Helped |
| Preserve caption order rather than randomly shuffling captions (ablation) | COSA · DiDeMo | R@1: 50.9 -> 54.8 | Helped |
| Replace vanilla MLM and GM with CMLM and CGM (ablation) | COSA training objectives · DiDeMo | R@1: 49.2 -> 53.0 | Helped |
| Replace vanilla ITC and ITM with CITC and CITM (ablation) | COSA training objectives · DiDeMo | R@1: 53.0 -> 52.3 | Hurt |
| Select samples randomly instead of by vision-feature similarity (ablation) | COSA sample concatenation · DiDeMo | R@1: 53.7 -> 54.8 | Helped |
| Select samples randomly instead of by text-feature similarity (ablation) | COSA sample concatenation · DiDeMo | R@1: 48.2 -> 54.8 | Helped |
| Use four sampled frames rather than one (ablation) | SST with CLIP-B on WebVid2.5M · DiDeMo | R@1: 43.6 -> 51.4 | Helped |
Surprising findings
- Concatenating randomly sampled, semantically unrelated samples works better than selecting similar samples; similarity-based grouping lowers DiDeMo R@1 from 54.8 to 53.7 (vision) or 48.2 (text).
- Contrary to Singularity, sampling multiple frames improves downstream performance (except VIST), though COSA remains more effective.
Limitations
- Concatenated samples have weak clip relevance and rapid scene changes, so the pseudo videos are not realistic.
- The method was only validated in vision-language pretraining and not in other domains.
- The largest model relies on captions regenerated by an additionally trained captioner.
Experimental setup
Datasets: CC3M CC12M COCO SBU VG WebVid2.5M LAION-102M CC4M MSRVTT DiDeMo LSMDC ActivityNet
Benchmarks: MSRVTT DiDeMo LSMDC ActivityNet MSRVTT-QA MSVD-QA TGIF-QA ActivityNet-QA MSVD VATEX YouCook2 TVC
Metrics: Recall@1 Recall@5 Recall@10 accuracy BLEU@4 CIDEr SPICE
Baselines: Singularity VINDLU LAVENDER OmniVL HiTeA LF-VILA GIT GIT2 BLIP-2 BEiT-3 mPLUG-2 VALOR
Builds on: BLIP ALBEF Frozen Singularity
Closest papers by meaning
- Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization (ICLR 2024 · similarity 0.88)
- CogVLM: Visual Expert for Pretrained Language Models (NeurIPS 2024 · similarity 0.87)
- Concept-Aware Batch Sampling Improves Language-Image Pretraining (CVPR 2026 · similarity 0.87)
- Mostly Text, Smart Visuals: Asymmetric Text-Visual Pruning for Large Vision-Language Models (CVPR 2026 · similarity 0.87)
- PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding (NeurIPS 2025 · similarity 0.87)
- Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.