Research map / Label-efficient and robust learning / Knowledge distillation
CustomKD: Customizing Large Vision Foundation for Edge Model Improvement via Knowledge Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CustomKD, which alternates between customizing a large vision foundation model's (DINOv2/OpenCLIP) features with the frozen student head and distilling those features into edge models for UDA and SSL image classification.
Problem
Knowledge distillation from large vision foundation models (ViT-L teachers) to small edge CNNs yields limited gains because of large capacity and architecture discrepancies between teacher and student.
Approach
Alternate between a feature customization stage, where a teacher projection layer is trained through the frozen student classifier to map teacher features into the student's representation space, and a KD stage where the student imitates both the original teacher features and the customized features via MSE losses, alongside CE/entropy-minimization losses on labeled and unlabeled data.
Key results
- On OfficeHome UDA with MobileNetV3 student and DINOv2 teacher, DKD+CustomKD reaches 74.63% average accuracy versus 69.48% for DKD alone.
- On CIFAR-100 with 400 labels, AdaMatch+CustomKD achieves 32.51% error rate, versus 52.07% for AdaMatch (source).
- A WideResNet28-2 student distilled with CustomKD from DINOv2 ViT-L reaches 32.51% error, beating DINOv2 ViT-S linear probing at 44.69% with fewer parameters (1.48M vs 21.52M).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CustomKD to DKD | MobileNetV3 · OfficeHome UDA (Avg.) | accuracy: 69.48 -> 74.63 | Helped |
| Add CustomKD to DKD | MobileNetV3 · DomainNet UDA (Avg.) | accuracy: 37.99 -> 41.10 | Helped |
| Add CustomKD to TfFD | MobileNetV3 · OfficeHome UDA (clipart source; Avg.) | accuracy: 48.14 -> 57.96 | Helped |
| Add CustomKD to NORM | MobileNetV3 · OfficeHome UDA (clipart source; Avg.) | accuracy: 59.20 -> 66.65 | Helped |
| Add CustomKD to DisWot | MobileNetV3 · OfficeHome UDA (clipart source; Avg.) | accuracy: 77.12 -> 77.54 | Helped |
| Add L ˜ft alongside Lft (ablation) | CIFAR-100 SSL with WideResNet28-2 and DINOv2 ViT-L teacher · CIFAR-100 (400 labels) | error rate: 48.58 -> 32.51 | Helped |
| Add L ˜ft alongside Lft (ablation) | CIFAR-100 SSL with WideResNet28-2 and DINOv2 ViT-S teacher · CIFAR-100 (400 labels) | error rate: 49.88 -> 47.97 | Helped |
| Add L ˜ft alongside Lft (ablation) | CIFAR-100 SSL with WideResNet28-2 and OpenCLIP ViT-B teacher · CIFAR-100 (400 labels) | error rate: 58.46 -> 50.49 | Helped |
| Add L ˜ft alongside Lft (ablation) | CIFAR-100 SSL with WideResNet28-2 and OpenCLIP ViT-L teacher · CIFAR-100 (400 labels) | error rate: 60.89 -> 45.02 | Helped |
| Use the student head classifier instead of a randomly initialized teacher head classifier (ablation) | CustomKD with MobileNetV3 · OfficeHome UDA | accuracy: 47.15 -> 56.47 | Helped |
| Add Lft to LL and LU (ablation) | CustomKD with MobileNetV3 · OfficeHome UDA | accuracy: 56.16 -> 56.32 | No clear effect |
| Add L ˜ft to LL and LU (ablation) | CustomKD with MobileNetV3 · OfficeHome UDA | accuracy: 56.16 -> 62.14 | Helped |
Surprising findings
- Existing KD methods fail to improve students as much when teacher backbone is scaled from ViT-S to ViT-L, despite teacher gains.
- Using the student's head classifier for the teacher's customization yields better student accuracy even though teacher accuracy drops.
Limitations
- Evaluated mainly on image classification; dense prediction tasks such as segmentation are left for future work.
- Reported CIFAR-100 USB baseline results differ from the paper's own reproductions, indicating reproducibility variance.
Experimental setup
Datasets: OfficeHome DomainNet CIFAR-100 ImageNet
Benchmarks: OfficeHome DomainNet CIFAR-100 ImageNet
Metrics: classification accuracy error rate CKA
Baselines: FitNet Soft Target Logits DKD RKD CC DAN Deep Coral DANN DSAN TfFD NORM
Builds on: FitNet Decoupled KD USB SimMatch AdaMatch
Closest papers by meaning
- ScaleKD: Strong Vision Transformers Could Be Excellent Teachers (NeurIPS 2024 · similarity 0.87)
- Align-KD: Distilling Cross-Modal Alignment Knowledge for Mobile Vision-Language Large Model Enhancement (CVPR 2025 · similarity 0.87)
- Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity (NeurIPS 2025 · similarity 0.86)
- FreeKD: Knowledge Distillation via Semantic Frequency Prompt (CVPR 2024 · similarity 0.86)
- Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learning (ICLR 2025 · similarity 0.86)
- Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.