Research map / Vision-language and multimodal / CLIP and zero-shot
ViTamin: Designing Scalable Vision Models in the Vision-Language Era
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Benchmarks ViT, ConvNeXt and CoAtNet under CLIP training on DataComp-1B and proposes ViTamin, a hybrid MBConv+Transformer vision encoder for VLMs, plus Locked-Text Tuning, reaching 82.9% ImageNet zero-shot with ViTamin-XL.
Problem
Vanilla ViTs remain the default image encoder for vision-language models, yet architecture choices from ImageNet-era research (ConvNets, hybrids) have rarely been evaluated under CLIP-style pretraining at scale.
Approach
Re-benchmark pure ViT, pure ConvNet and hybrid vision backbones under CLIP on DataComp-1B across data and model scales, then design a 3-stage hybrid (two MBConv-LN stages plus a GeGLU Transformer stage, output stride 16) tailored for VLMs, and freeze a pretrained text encoder to train smaller image encoders (Locked-Text Tuning).
Key results
- ViTamin-L@256 reaches ImageNet zero-shot accuracy 2.0% above OpenCLIP ViT-L/14@224 under identical DataComp-1B training.
- ViTamin-XL with 436M parameters attains 82.9% ImageNet zero-shot accuracy, above EVA-02-CLIP-E/14 at 82.0% with 4.4B parameters.
- Locked-Text Tuning improves ViTamin-S by +4.0% and ViTamin-B by +4.9% without extra cost, and ViTamin-B-LTT reaches +23.3% over the ViT-B/32 baseline on the DataComp small-scale track.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ViTamin-L instead of ViT-L/14 with matched training data and scheme | OpenCLIP on DataComp-1B · ImageNet zero-shot | accuracy: 79.2 -> 81.2 | Helped |
| Scale ViTamin to XL and train with 40.0B seen samples | ViTamin · ImageNet zero-shot | accuracy: 82.9 vs 82.0 | Helped |
| Increase training seen samples | ViT, ConvNeXt, and CoAtNet · ImageNet zero-shot | accuracy: 45.8% to 60.0% (+14.2%) and 65.6% (+19.8) | Helped |
| Scale model parameters from base to large | ViT · ImageNet zero-shot | accuracy: +6.4% at 512M samples; +6.3% at 1.28B samples | Helped |
| Use smaller ViT patch sizes for higher feature resolution | ViT · ImageNet zero-shot | accuracy | Helped |
| Use CoAtNet rather than ViT or ConvNeXt | CLIP vision backbone · ImageNet zero-shot | accuracy: +2.9% and +3.2% | Helped |
| Use ViTamin-L rather than ViT-L/14 | F-ViT open-vocabulary detector · OV-LVIS | APr: +3.1% | Helped |
| Use ViTamin-L rather than ViT-L/14 | FC-CLIP open-vocabulary segmentation · average 8 datasets | segmentation score: +2.6% | Helped |
| Add Locked-Text Tuning with a frozen pretrained text encoder | ViTamin-S and ViTamin-B · ImageNet zero-shot | accuracy: +4.0% for the small variant and +4.9% for the base variant | Helped |
| Add Locked-Text Tuning | ViTamin-B · ImageNet zero-shot, 128M seen samples | accuracy: 42.2 -> 52.7 | Helped |
| Add Locked-Text Tuning | ViTamin-B · average 38 datasets, 128M seen samples | accuracy: 38.3 -> 47.2 | Helped |
| Increase image size from 336 to 384 | ViTamin-L in LLaVA-1.5 · VizWiz | score: 51.1 -> 55.4 | Helped |
Surprising findings
- ConvNeXt, which often beats ViT/16 on ImageNet, falls behind ViT/16 under CLIP training on DataComp-1B, reversing the ImageNet-era trend.
- Improving the vision architecture (ViTamin-B, +12.8%) gives far larger gains than the top DataComp filtering solution (+2.3%) on the same dataset size.
Limitations
- CoAtNet-4 is hard to scale to billions of samples due to GPU memory use.
- Long-schedule ViTamin-L and XL training uses 184 to 312 A100 GPUs, limiting reproducibility for smaller labs.
- Downstream detection and segmentation use frozen backbones with sliding-window inference, so results depend on those frameworks.
Experimental setup
Datasets: DataComp-1B
Benchmarks: ImageNet DataComp 38-dataset suite OV-COCO OV-LVIS ADE Cityscapes MV A-150 A-847 PC-459 PC-59 PAS-21
Metrics: zero-shot accuracy average accuracy over 38 datasets AP PQ mIoU LMM benchmark scores
Baselines: ViT-L/14 (OpenCLIP) ViT-L/14 (CLIPA-v2) ViT-H/14 (CLIPA-v2) EVA-02-CLIP-E/14 SigLIP ViT-L/16 ConvNeXt-L CoAtNet
Builds on: CLIP CoAtNet Vision Transformer OpenCLIP LLaVA-1.5
Closest papers by meaning
- Bridge the Modality and Capability Gaps in Vision-Language Model Selection (NeurIPS 2024 · similarity 0.88)
- Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models (CVPR 2025 · similarity 0.88)
- From Pixels to Words -- Towards Native Vision-Language Primitives at Scale (ICLR 2026 · similarity 0.87)
- VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models (ICML 2026 · similarity 0.87)
- Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distance (CVPR 2026 · similarity 0.87)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.