Research map / Vision-language and multimodal / CLIP and zero-shot
Learning Vision from Models Rivals Learning Vision from Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns visual representations purely from synthetic data: Llama-2 synthesizes captions, Stable Diffusion 1.5 renders 4 images per caption, and a StableRep-style multi-positive contrastive loss plus iBOT masked image modeling is trained on 600M synthetic images.
Problem
Visual representation learning normally requires large curated real image datasets, which are costly to collect; the paper asks whether purely synthetic images from generative models can train representations that rival those learned from real data.
Approach
Treat each synthetic caption as a visual class: generate multiple images per caption with a text-to-image diffusion model and align them via multi-positive contrastive learning combined with iBOT masked image modeling, with LLM-based caption synthesis (in-context templates, GPT-4-proposed backgrounds) providing fine-grained, scalable concept coverage.
Key results
- SynCLR ViT-B/16 and ViT-L/14 reach 80.7% and 83.0% ImageNet linear probing, on par with OpenAI CLIP (80.2% and 83.9%) and below DINO v2 (83.9%/85.7%).
- On ADE20k segmentation with UperNet, SynCLR ViT-B reaches 54.3 mIoU, 6.2 above MAE-style ImageNet pretraining (48.1) and 4.3 above iBOT (50.0).
- Multi-positive caption-level contrastive learning outperforms supervised cross-entropy by 3.4% and SimCLR by 11.7% on ImageNet linear evaluation under matched data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the full synthetic caption configuration instead of cc12m captions (ablation) | SynCLR · ImageNet-1K linear evaluation | top-1 accuracy: 77.1 -> 78.8 | Helped |
| Use the full synthetic caption configuration instead of cc12m captions (ablation) | SynCLR · Fine-grained classification average (9 datasets) | average accuracy: 85.3 -> 88.1 | Helped |
| Use GPT-generated backgrounds instead of Places backgrounds in LLM captions (ablation) | SynCLR caption synthesis · Fine-grained classification average (9 datasets) | average accuracy: 81.8 -> 81.9 | No clear effect |
| Add EMA to the StableRep baseline (ablation) | SynCLR · ImageNet-1K linear evaluation | top-1 accuracy: 75.8 -> 76.7 | Helped |
| Add iBOT objective (ablation) | SynCLR · ADE20K semantic segmentation | mIoU: 49.5 -> 50.5 | Helped |
| Add multi-crop strategy (ablation) | SynCLR · ImageNet-1K linear evaluation | top-1 accuracy: 76.7 -> 78.6 | Helped |
| Change classifier-free guidance scale from 2 to 3 (ablation) | SynCLR image generation · ImageNet-1K linear evaluation | top-1 accuracy: 72.8 -> 72.6 | No clear effect |
| Define positive classes by captions rather than individual images (SimCLR) | SynCLR representation learning · ImageNet-1K linear evaluation | top-1 accuracy: 63.6 -> 75.3 | Helped |
| Define classes by captions rather than fixed balanced ImageNet classes (supervised CE) | SynCLR representation learning · ImageNet-1K linear evaluation | top-1 accuracy: 71.9 -> 75.3 | Helped |
| Train with synthetic rather than real images and captions | SynCLR ViT-B/16 · ImageNet-1K linear evaluation | top-1 accuracy: 80.7 vs 80.2 | Mixed |
| Train with synthetic rather than real data | SynCLR ViT-L/14 · Fine-grained classification average (9 datasets) | average accuracy: 90.4 vs 90.1 | No clear effect |
| Pre-train SynCLR rather than ImageNet-1K self-supervised methods | ViT-B semantic segmentation · ADE20K semantic segmentation | mIoU: +4.3 mIoU | Helped |
Surprising findings
- Naively asking the LLM to combine ImageNet and Places class names yields the worst caption-set performance, below simply using Places backgrounds, indicating synthetic captions must match the text-to-image model's training distribution.
Limitations
- Lags DINO v2 on ImageNet linear probing (3.2% for ViT-B, 2.7% for ViT-L), partly due to DINO v2's distillation from ViT-g.
- Finite diversity of generated images; relies on the text-to-image model's training distribution and concept coverage.
- Did not use higher-resolution training or other extras (SwiGLU, LayerScale) that could further improve results.
Experimental setup
Datasets: SynCaps-150M (synthetic captions) ImageNet-21k Places-365 ImageNet-1k cc12m ADE20k ImageNet linear eval fine-grained datasets (Aircraft, Cars, DTD, Flowers, Pets, SUN397, Caltech-101, Food-101, VOC2007)
Benchmarks: ImageNet linear probing ImageNet fine-tuning fine-grained classification (9 datasets) ADE20k semantic segmentation
Metrics: top-1 accuracy average accuracy mIoU
Baselines: CLIP OpenCLIP DINO v2 StableRep MAE iBOT SimCLR MoCo v3 BEiT v2
Builds on: StableRep iBOT MoCo v3 DINO
Closest papers by meaning
- Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Data (CVPR 2025 · similarity 0.88)
- ImagineFSL: Self-Supervised Pretraining Matters on Imagined Base Set for VLM-based Few-shot Learning (CVPR 2025 · similarity 0.87)
- Can Generative Models Improve Self-Supervised Representation Learning? (AAAI 2025 · similarity 0.86)
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.86)
- Retrieval-Enhanced Contrastive Vision-Text Models (ICLR 2024 · similarity 0.86)
- Learning Visual Generative Priors without Text (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.