Research map / Architectures and efficiency / Attention and transformers
ConvNet vs Transformer, Supervised vs CLIP: Beyond ImageNet Accuracy
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Compares ConvNeXt and ViT, each under supervised and OpenCLIP training, on properties beyond ImageNet accuracy (mistakes, shape/texture bias, calibration, robustness, transferability, synthetic data, invariance) using models with matched ImageNet accuracy.
Problem
ImageNet accuracy alone fails to distinguish models with similar accuracy, which differ in error types, calibration, transferability, and invariance, so model selection based on it can be misleading.
Approach
Fix the comparison to four base-sized models (ConvNeXt-B and DeiT3/ViT-B, each supervised and CLIP on LAION-400M) with near-identical ImageNet accuracy, and evaluate many off-ImageNet properties without further training, plus size-scaling experiments in the appendix.
Key results
- CLIP models have lower ImageNet-X error ratios (about 1.19-1.33) than supervised models (about 1.43-1.50) despite lower ImageNet accuracy.
- Supervised ConvNeXt-B reaches ECE 0.025 on ImageNet-1K versus 0.043 for supervised ViT-B, and linear-probe VTAB transfer close to CLIP models (both averaging the same).
- ViT-CLIP Base reaches 46.4% shape bias versus 33.3% for ConvNeXt-sup Base (ConvNeXt-CLIP Base 45.9%); ConvNeXt outperforms ViT on PUG-ImageNet in 6 of 7 comparisons.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CLIP training instead of supervised training | ConvNeXt and ViT · ImageNet-X | error ratio: 1.43 -> 1.23; 1.45 -> 1.19 | Helped |
| Use ConvNeXt instead of ViT | supervised models · PUG-ImageNet | accuracy: 29.4 -> 32.2 | Helped |
| Use supervised training instead of CLIP training | ConvNeXt and ViT · PUG-ImageNet | accuracy: 27.0 -> 32.2; 25.2 -> 29.4 | Helped |
| Use ConvNeXt instead of ViT | supervised models · ImageNet-1K | ECE: 0.043 -> 0.025 | Helped |
| Use ConvNeXt instead of ViT | supervised models · ImageNet-R | ECE: 0.041 -> 0.028 | Helped |
| Use supervised training instead of CLIP training | ConvNeXt and ViT · ImageNet-1K | ECE: 0.085 -> 0.025; 0.073 -> 0.043 | Helped |
| Use ViT instead of ConvNeXt | supervised models · Shape-texture cue-conflict | shape bias: 33.3 -> 39.8 | Helped |
| Use CLIP training instead of supervised training | ConvNeXt and ViT · Shape-texture cue-conflict | shape bias: Shape bias in CLIP models improved by 7% and 12% | Helped |
| Use ConvNeXt instead of ViT | supervised models · transformation invariance | accuracy under transformations | Helped |
| Use ConvNeXt instead of ViT | supervised models · VTAB | linear-probe accuracy | Helped |
| Use CLIP training instead of supervised training | models evaluated on ImageNet robustness benchmarks · ImageNet variants | accuracy | Hurt |
| Use ConvNeXt instead of ViT | models evaluated on synthetic data · PUG-ImageNet | accuracy: ConvNeXt is better in 6 out of 7 comparisons. | Helped |
Surprising findings
- CLIP models are overconfident in calibration (higher ECE) even though they are more accurate on ImageNet-R, suggesting training data and objectives matter more than architecture.
- Supervised ConvNeXt-B achieves higher accuracy at 336px than at its native 224px resolution.
Limitations
- Uses publicly available pretrained checkpoints, so training data size and quality cannot be controlled.
- Restricted to properties evaluable without further training; downstream tasks like detection are excluded.
- Main comparison is limited to base-sized models with one CLIP data source (LAION-400M).
Experimental setup
Datasets: ImageNet-1K ImageNet-X ImageNet-R cue-conflict PUG-ImageNet VTAB ImageNet-V2 ImageNet-A ImageNet-C ImageNet-Sketch ImageNet-Real ImageNet-Hard
Benchmarks: ImageNet-X ImageNet-R cue-conflict ImageNet-V2 ImageNet-A ImageNet-C ImageNet-Sketch ImageNet-Real ImageNet-Hard VTAB PUG-ImageNet ImageNet-1K
Metrics: error ratio top-1 accuracy shape bias texture bias expected calibration error linear probing accuracy
Baselines: ViT-sup ConvNeXt-sup ViT-clip ConvNeXt-clip
Builds on: ConvNeXt Vision Transformer CLIP DeiT III VTAB
Closest papers by meaning
- A ConvNet for the 2020s (CVPR 2022 · similarity 0.90)
- Do Better ImageNet Models Transfer Better? (CVPR 2019 · similarity 0.88)
- Vision Transformers Are Robust Learners (AAAI 2022 · similarity 0.88)
- Does Robustness on ImageNet Transfer to Downstream Tasks? (CVPR 2022 · similarity 0.88)
- DECO: Unleashing the Potential of ConvNets for Query-based Detection and Segmentation (ICLR 2025 · similarity 0.86)
- Synergy and Diversity in CLIP: Enhancing Performance Through Adaptive Backbone Ensembling (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.