Research map / Label-efficient and robust learning / Self-supervised and contrastive
Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use supervised ConvNeXt-Base pretrained on ImageNet-21k | backbone selection across evaluated tasks · classification, detection, segmentation, retrieval, and OOD evaluation | aggregate performance ranking | Helped |
| Use supervised SwinV2-Base pretrained on ImageNet-21k and fine-tuned on ImageNet-1k | backbone selection for classification · classification across datasets and settings | aggregate performance ranking | Helped |
| Use supervised ConvNeXt-Base rather than vanilla ViT backbones | computer vision backbone selection · most evaluated tasks | relative performance | Helped |
| Use SSL pretraining rather than supervised pretraining at a similar pretraining-data scale | backbone pretraining · classification and retrieval | performance | Helped |
| Use supervised pretraining rather than SSL pretraining | backbones for detection and segmentation · detection and segmentation | performance | Helped |
| Increase model parameter count | ViT and CNN backbones · across evaluated tasks | relative performance: spearman ρ = 0.58; spearman ρ = 0.35 | Helped |
| Increase pretraining dataset size | ViT and CNN backbones · across evaluated tasks | relative performance: ρ = 0.72; ρ=0.33 | Helped |
| End-to-end fine-tune rather than freeze the backbone and train only the head | ViT backbones · dense prediction tasks | relative performance | Helped |
| Use CLIP pretraining rather than vanilla ViT alternatives | ViT backbones · almost all evaluated tasks except OOD detection | performance | Helped |
| Use generative pretraining such as MAE or Stable Diffusion rather than supervised or contrastive pretraining | backbone pretraining · evaluated computer vision tasks | performance | Hurt |
| Use supervised pretraining rather than SSL pretraining | backbones with matched architectures · ImageNet adversarial robustness | adversarial robustness | Hurt |
| Remove MiDaS and MAE backbones from the retrieval-versus-classification comparison (ablation) | cross-task performance correlation analysis · retrieval and classification | Spearman correlation: ρ = 0.49; ρ increased to 0.8 | Helped |
Closest papers by meaning
- Synergy and Diversity in CLIP: Enhancing Performance Through Adaptive Backbone Ensembling (ICLR 2025 · similarity 0.90)
- Beyond Random Augmentations: Pretraining with Hard Views (ICLR 2025 · similarity 0.87)
- ScaleKD: Strong Vision Transformers Could Be Excellent Teachers (NeurIPS 2024 · similarity 0.86)
- Dataset Distillation for Pre-Trained Self-Supervised Vision Models (NeurIPS 2025 · similarity 0.86)
- UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling (NeurIPS 2024 · similarity 0.86)
- ConvNet vs Transformer, Supervised vs CLIP: Beyond ImageNet Accuracy (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.