Research map / Vision-language and multimodal / CLIP and zero-shot
COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes COSMOS, a cross-modality self-distillation framework for CLIP-style vision-language pre-training with text cropping of long synthetic captions and a cross-attention module, pre-trained on 3M-30M pairs.
Problem
Contrastive VLM pre-training (CLIP) uses global matching that suppresses non-dominant image content and treats captions as bags of words, hurting dense prediction and compositional understanding.
Approach
Applies DINO-style multi-crop self-distillation to both images and texts (text cropping by sentence sampling from long synthetic captions), and adds cross-attention modules in the student so the image [cls] and text [eot] embeddings are conditioned on the other modality before distilling to the teacher's tokens.
Key results
- On MSCOCO zero-shot retrieval, COSMOS pre-trained on Merged-30M reaches 68.0% I2T and 52.5% T2I R@1, beating Llip (2.5B pairs) at 63.4% and 45.6%.
- With CC3M pre-training, COSMOS reaches 37.1% ImageNet zero-shot accuracy versus 23.9% for CLIP and 25.9% for SigLIP.
- COSMOS trained on 30M samples averages 20.0% mIoU across segmentation benchmarks versus 16.5% for OpenCLIP trained on 1B samples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use COSMOS instead of Llip for pre-training on the reported datasets | ViT-B/16 vision-language model · MSCOCO | I2T R@1: 63.4 -> 68.0 | Helped |
| Use COSMOS instead of Llip for pre-training on the reported datasets | ViT-B/16 vision-language model · MSCOCO | T2I R@1: 45.6 -> 52.5 | Helped |
| Use COSMOS rather than DreamLIP with Merged-30M pre-training | ViT-B/16 vision-language model · ImageNet | top-1 accuracy: 58.4 -> 57.6 | Hurt |
| Use COSMOS rather than OpenCLIP trained on 1B samples | ViT-B/16 vision-language model · Semantic segmentation (average across benchmarks) | mIoU: 16.5 -> 20.0 | Helped |
| Apply SCLIP's correlative self-attention mechanism | COSMOS · Semantic segmentation (average across benchmarks) | mIoU: 20.0 -> 37.8 | Helped |
| Use COSMOS rather than DreamLIP trained on the same 30M data | ViT-B/16 vision-language model · SugarCrepe | average accuracy: 81.8 -> 86.6 | Helped |
| Use COSMOS rather than OpenCLIP trained on 1B samples | ViT-B/16 vision-language model · MMVP-VLM | accuracy: 25.9 -> 25.9 | No clear effect |
| Add image augmentation (ablation) | CLIP trained on CC3M · ImageNet | top-1 accuracy: 17.5 -> 19.1 | Helped |
| Add EMA (ablation) | CLIP with image augmentation trained on CC3M · ImageNet | top-1 accuracy: 19.1 -> 19.1 | No clear effect |
| Add image self-distillation (ablation) | CLIP with image augmentation and EMA, trained on CC3M · ImageNet | top-1 accuracy: 19.1 -> 22.8 | Helped |
| Add long synthetic-caption text augmentation (ablation) | CLIP with image self-distillation, trained on CC3M · ImageNet | top-1 accuracy: 22.8 -> 34.4 | Helped |
| Add text self-distillation (ablation) | CLIP with text augmentation, trained on CC3M · ImageNet | top-1 accuracy: 34.4 -> 35.4 | Helped |
Surprising findings
- COSMOS trained on 30M samples matches OpenCLIP trained on 1B samples on MMVP-VLM (25.9%).
Limitations
- Relies on long synthetic captions generated by MLLMs for text augmentation.
- Adding self-distillation without CLIP loss scaling can underperform; the paper's loss design is tuned for its setup.
- Fine-grained ImageNet/fine classes still lag some baselines such as Aircraft on several settings.
Experimental setup
Datasets: CC3M CC12M YFCC15M Merged-30M DreamLIP synthetic long captions
Benchmarks: Flickr30K MSCOCO ImageNet Food-101 CIFAR-10 CIFAR-100 SUN397 Cars Aircraft DTD Pets Caltech-101
Metrics: R@1 R@5 R@10 top-1 accuracy mIoU
Baselines: CLIP SigLIP DreamLIP LaCLIP MLLM-A SLIP MaskCLIP SF-CLIP OpenCLIP MetaCLIP Llip SILC
Builds on: CLIP DINO SLIP SILC DreamLIP
Closest papers by meaning
- Enhancing Large Vision Language Models with Self-Training on Image Comprehension (NeurIPS 2024 · similarity 0.88)
- LoMo: Local Modality Substitution for Deeper Vision-Language Fusion (NeurIPS 2026 · similarity 0.87)
- Data Selection for Fine-tuning Vision Language Models via Cross Modal Alignment Trajectories (ICML 2026 · similarity 0.87)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.87)
- TiMix: Text-Aware Image Mixing for Effective Vision-Language Pre-training (AAAI 2024 · similarity 0.87)
- Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.