Research map / Label-efficient and robust learning / Knowledge distillation
CLIP-KD: An Empirical Study of CLIP Model Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirical study of distilling small CLIP student models from a large teacher CLIP using relation, feature, gradient, and contrastive distillation losses, with a unified FD+ICL+CRD recipe that works across different architectures.
Problem
Improving small CLIP models for resource-constrained use via knowledge distillation from a large teacher CLIP, without requiring matched teacher-student architectures as TinyCLIP's weight inheritance does.
Approach
Proposes several CLIP-specific distillation losses (contrastive relational KL, feature MSE mimicry, masked feature MSE, gradient matching, interactive contrastive learning where the student anchors against teacher embeddings, and augmented fused-feature contrast) and finds plain feature MSE plus interactive contrastive learning, combined with relational distillation, works best for any teacher-stu
Key results
- Feature distillation with MSE improves ViT-T/16 student by 3.6% zero-shot ImageNet top-1 over baseline on CC3M+12M, the best single loss.
- With ViT-L/14 teacher pretrained on LAION-400M, CLIP-KD reaches 57.5% zero-shot ImageNet top-1 for ViT-B/16 and 55.4% for ResNet-50, surpassing the no-KD baseline by 20.5% and 20.1%.
- CLIP-KD outperforms TinyCLIP by 1.8% ImageNet accuracy and 1.7% cross-modal retrieval on average, while also supporting cross-architecture distillation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add feature distillation using MSE (ablation) | ViT-T/16 CLIP · ImageNet-1K | accuracy: 30.6 -> 34.2 | Helped |
| Add masked feature distillation (ablation) | ViT-T/16 CLIP · ImageNet-1K | accuracy: 30.6 -> 34.1 | Helped |
| Add contrastive relational distillation (ablation) | ViT-T/16 CLIP · ImageNet-1K | accuracy: 30.6 -> 31.9 | Helped |
| Add gradient distillation (ablation) | ViT-T/16 CLIP · ImageNet-1K | accuracy: 30.6 -> 31.5 | Helped |
| Add interactive contrastive learning (ablation) | ViT-T/16 CLIP · ImageNet-1K | accuracy: 30.6 -> 33.1 | Helped |
| Add augmented feature distillation (ablation) | ViT-T/16 CLIP · ImageNet-1K | accuracy: 30.6 -> 31.4 | Helped |
| Combine feature distillation and interactive contrastive learning (ablation) | ViT-T/16 CLIP · ImageNet-1K | accuracy: 34.2 -> 34.6 | Helped |
| Add CRD to FD+ICL (ablation) | ViT-T/16 CLIP · ImageNet-1K | accuracy: 34.6 -> 34.9 | Helped |
| Add gradient distillation to FD+ICL+CRD (ablation) | ViT-T/16 CLIP · ImageNet-1K | accuracy: 34.9 -> 34.8 | Hurt |
| Add augmented feature distillation to FD+ICL+CRD (ablation) | ViT-T/16 CLIP · ImageNet-1K | accuracy: 34.9 -> 34.8 | Hurt |
| Distill ViT-B/16 from ViT-L/14 pretrained on Laion-400M | ViT-B/16 · ImageNet-1K | accuracy: 37.0 -> 57.5 | Helped |
| Distill ResNet-50 from ViT-B/16 pretrained on Laion-400M | ResNet-50 · ImageNet-1K | accuracy: 35.3 -> 55.4 | Helped |
Surprising findings
- A simple feature MSE mimicry loss outperforms more elaborate relational, gradient, and contrastive distillation methods.
- The more capable ViT-L/14 teacher underperforms the weaker ViT-B/16 teacher when distilling a ViT-T/16 student, attributed to capacity gap.
Limitations
- Distillation uses only final output embeddings, not intermediate hidden-layer information.
- Large teacher and small student may suffer capacity gap, hurting distillation (e.g., ViT-L/14 underperforms ViT-B/16 teacher).
- Experiments are limited to CC3M/CC12M scale for most results.
Experimental setup
Datasets: CC3M CC12M LAION-400M ImageNet ImageNet-V2 ImageNet-R ImageNet-Sketch MSCOCO Flickr30K
Benchmarks: ImageNet ImageNet-V2 ImageNet-R ImageNet-Sketch CC3M validation MSCOCO Flickr30K
Metrics: top-1 accuracy R@1 image-to-text R@1 text-to-image
Baselines: TinyCLIP CLIP without distillation (baseline) CRD FD MFD GD ICL AFD
Builds on: CLIP TinyCLIP DistillVLM MAE
Closest papers by meaning
- Building Vision-Language Models on Solid Foundations with Masked Distillation (CVPR 2024 · similarity 0.90)
- Meta CLIP 2: A Worldwide Scaling Recipe (NeurIPS 2025 · similarity 0.90)
- Demystifying CLIP Data (ICLR 2024 · similarity 0.90)
- A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation (ICLR 2024 · similarity 0.90)
- Retaining Knowledge and Enhancing Long-Text Representations in CLIP through Dual-Teacher Distillation (CVPR 2025 · similarity 0.90)
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.