Research map / Label-efficient and robust learning / Knowledge distillation
Accessing Vision Foundation Models via ImageNet-1K
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proteus distills large vision foundation models (DINOv2, SynCLR, CLIP) into randomly initialized ViTs using only ImageNet-1K (1.2M images) via token, patch, and feature-level objectives without one-hot labels or the classification head.
Problem
Replicating the capabilities of vision foundation models (e.g., DINOv2, CLIP, SynCLR) is hard because their training data is private and training is extremely costly; the question is whether task-agnostic compression can work with small public data like ImageNet-1K.
Approach
Remove label-based CE loss and pre-classification-head distillation (which cause dataset bias), and distill the teacher with three objectives: token-level MSE on the CLS token, feature-level MSE on the final features, and masked patch-level MSE on teacher patch tokens (iBOT-style), using a randomly initialized student.
Key results
- Proteus-L/14 distilled from DINOv2-g/14 on 1.2M ImageNet images matches DINOv2-L/14 (142M images) on average fine-grained accuracy (91.0 vs 91.0) and ImageNet linear probing is 86.2 vs 86.3.
- ViT-S/14 Proteus distilled from DINOv2-B reaches 81.8% ImageNet linear accuracy and 85.8% fine-grained average, surpassing CLIP-B/32 (82.5 avg is CLIP; Proteus-S is 85.8) and outperforming DeiT ViT-S/16 (81.2 ImageNet, 77.8 fine-grained).
- Proteus-B/14 distilled from DINOv2-L reaches 84.9% ImageNet and 88.6% fine-grained average, beating SynCLR-B/16 (88.8 average is similar) and CLIP-B/16 (85.8).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| distill DINOv2 into ViT-B/14 using ImageNet-1K | Proteus · ImageNet-1K linear evaluation | accuracy: 83.4 -> 84.9 | Helped |
| distill DINOv2 into ViT-B/14 using ImageNet-1K | Proteus · 12 fine-grained datasets | average accuracy: 88.8 -> 88.6 | Mixed |
| distill DINOv2 into ViT-L/14 using ImageNet-1K | Proteus · ImageNet-1K linear evaluation | accuracy: 86.3 -> 86.2 | Hurt |
| distill SynCLR into ViT-B/14 using ImageNet-1K | Proteus · ADE20K | mIoU: 53.4 -> 52.5 | Hurt |
| train on ImageNet-Merge rather than ImageNet-1K (ablation) | Proteus-B/14 distilled from CLIP-L/14 · 12 fine-grained datasets | average accuracy: 85.7 -> 87.8 | Helped |
| train on ImageNet-Single generated from one image rather than ImageNet-1K (ablation) | Proteus ViT-S/14 · 12 fine-grained datasets | average accuracy: 85.8 -> 68.7 | Hurt |
| initialize the student by inheriting teacher weights (ablation) | Proteus ViT-S/14 · 12 fine-grained datasets | average accuracy: 85.8 -> 85.3 | Hurt |
| use a Linear+GELU projector instead of LN+Linear (ablation) | Proteus ViT-S/14 · ImageNet-1K linear evaluation | accuracy: 81.7 -> 81.8 | No clear effect |
| use a GELU+Linear projector instead of LN+Linear (ablation) | Proteus ViT-S/14 · 12 fine-grained datasets | average accuracy: 85.3 -> 85.3 | No clear effect |
| use a larger DINOv2-g/14 teacher instead of DINOv2-B/14 (ablation) | Proteus ViT-S/14 · 12 fine-grained datasets | average accuracy: 85.8 -> 80.6 | Hurt |
| use a DINOv2-L/14 teacher instead of DINOv2-B/14 (ablation) | Proteus ViT-S/14 · 12 fine-grained datasets | average accuracy: 85.8 -> 82.2 | Hurt |
Surprising findings
- Using a larger teacher (DINOv2-g) yields lower fine-grained average (80.6) than DINOv2-B (85.8) for ViT-S students, attributed to dimension-projection information loss.
- Weight inheritance from the teacher's pretrained weights degrades performance, so a randomly initialized student is used.
Limitations
- Only validated on ImageNet-1K; efficacy at larger data scales unverified due to compute constraints.
- Student must keep the same patch size as the teacher because of patch and feature-level objectives.
- Lags DINOv2 on ImageNet-A and on some fine-grained datasets (FGVC-Aircraft, Stanford Cars, SUN397).
Experimental setup
Datasets: ImageNet-1K ImageNet-Merge ImageNet-Single ADE20K NYU-Depth V2 12 fine-grained classification datasets
Benchmarks: ImageNet-1K Aircraft Caltech101 Cars CIFAR-10 CIFAR-100 DTD Flowers Food-101 Pets SUN397 VOC2007
Metrics: top-1 accuracy mIoU RMSE robustness accuracy
Baselines: DeiT DINOv2 CLIP OpenCLIP SynCLR DINO iBOT
Builds on: DeiT DINOv2 iBOT TinyCLIP
Closest papers by meaning
- Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models (ICML 2024 · similarity 0.88)
- ScaleKD: Strong Vision Transformers Could Be Excellent Teachers (NeurIPS 2024 · similarity 0.86)
- How Well Does GPT-4o Understand Vision? Evaluating Multimodal Foundation Models on Standard Computer Vision Tasks (ICLR 2026 · similarity 0.85)
- Specialized Foundation Models Struggle to Beat Supervised Baselines (ICLR 2025 · similarity 0.85)
- VESSA: Video-based objEct-centric Self-Supervised Adaptation for Visual Foundation Models (NeurIPS 2025 · similarity 0.85)
- CustomKD: Customizing Large Vision Foundation for Edge Model Improvement via Knowledge Distillation (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.