Research map / Label-efficient and robust learning / Knowledge distillation
Hybrid Distillation: Connecting Masked Autoencoders with Contrastive Learners
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Hybrid Distill, which distills a MIM teacher's token relations in the last-but-one layers and a supervised/CL teacher's final-layer features into a ViT student, with progressive redundant token masking to cut cost.
Problem
Single-teacher feature distillation for representation learning trades off discrimination against attention diversity; asymmetric designs that increase diversity may harm discrimination.
Approach
Distill from two teachers at different locations and with different targets: token relations (attention) from a MIM teacher in layers L-1 and L-2 to preserve diversity, and output features from a supervised/CL teacher at the last layer to preserve discrimination, plus a progressive redundant token masking guided by the MIM teacher.
Key results
- Hybrid Distill with ViT-B and MAE+CLIP teachers reaches 85.1 ImageNet-1K top-1, 50.6 AP box and 44.4 AP mask on COCO, and 51.5 mIoU on ADE20K, outperforming Distill-CLIP by 1.1, 0.9 and 1.2.
- With ViT-L, Hybrid Distill reaches 88.0 IN-1K, 54.6 AP box, 47.6 AP mask and 56.3 mIoU ADE20K.
- 300-epoch Hybrid Distill with Cascade Mask-RCNN (MAE+CLIP) achieves 53.4 AP box, outperforming 1600-epoch dBOT-CLIP's 53.6 only slightly while exceeding dBOT-DeiT and dBOT-MAE at 2x800 epochs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine DeiT feature distillation with MAE token-relation distillation (ablation) | Hybrid Distill with ViT-B · COCO | box AP: 47.5 -> 50.0 | Helped |
| Combine CLIP feature distillation with MAE token-relation distillation (ablation) | Hybrid Distill with ViT-B · COCO | box AP: 49.1 -> 50.4 | Helped |
| Use MAE token relations rather than MAE feature maps as the distillation target (ablation) | Hybrid Distill with DeiT and MAE teachers · COCO | box AP: 49.6 -> 50.0 | Helped |
| Distill MAE relations at layers 10 and 11 rather than layers 1–11 (ablation) | Hybrid Distill with DeiT and MAE teachers · COCO | box AP: 48.8 -> 50.0 | Helped |
| Distill MAE relations at back layers 9, 10, and 11 rather than front layers 1, 2, and 3 (ablation) | Hybrid Distill with DeiT and MAE teachers · COCO | box AP: 47.4 -> 49.8 | Helped |
| Add progressive redundant-token masking at the 70% per-step setting (ablation) | Hybrid Distill with DeiT and MAE teachers · COCO | box AP: 50.0 -> 49.9 | Hurt |
| Use progressive rather than random token masking (ablation) | Hybrid Distill with DeiT and MAE teachers · COCO | box AP: 49.2 -> 49.9 | Helped |
| Use progressive rather than direct token masking (ablation) | Hybrid Distill with DeiT and MAE teachers · COCO | box AP: 49.6 -> 49.9 | Helped |
| Increase the MAE supervision weight from alpha 0 to alpha 1.0 (ablation) | Hybrid Distill with DeiT and MAE teachers · COCO | box AP: 47.5 -> 50.0 | Helped |
| Increase the MAE supervision weight from alpha 0 to alpha 1.0 (ablation) | Hybrid Distill with CLIP and MAE teachers · COCO | box AP: 49.1 -> 50.4 | Helped |
| Add MAE token-relation supervision (ablation) | Hybrid Distill with DeiT and MAE teachers · COCO | mask AP: 41.8 -> 43.9 | Helped |
| Add MAE token-relation supervision (ablation) | Hybrid Distill with DeiT and MAE teachers · COCO | box AP: +2.5 AP^{box} | Helped |
Surprising findings
- Asymmetric decoders and encoder designs that increase attention diversity transfer teacher discrimination and harm it, and high-level mask reconstruction gives no diversity gain over direct feature distillation.
- Distilling the full token features of the MIM teacher performs worse than distilling its token relations, and naively distilling both teachers' final-layer features performs poorly.
Limitations
- Two teachers add overhead (about 1.2x training time for ViT-B versus a single teacher).
- Gains with CLIP are smaller than with DeiT, possibly due to the capacity gap between CLIP and MAE teachers.
- ImageNet-1K gains are small because distillation uses the same dataset.
Experimental setup
Datasets: ImageNet-1K ImageNet-21K COCO ADE20K CIFAR100 Stanford Cars iNaturalist19
Benchmarks: ImageNet-1K COCO ADE20K CIFAR100 Cars iNaturalist19
Metrics: top-1 accuracy AP box AP mask mIoU average head distance normalized mutual information
Baselines: DeiT MoCo v3 DINO MAE CAE SdAE CLIP FD dBOT Distill-DeiT Distill-MAE Distill-CLIP
Builds on: dBOT Feature Distillation (FD) MAE DeiT CLIP
Closest papers by meaning
- Contrastive Tuning: A Little Help to Make Masked Autoencoders Forget (AAAI 2024 · similarity 0.90)
- Asymmetric Masked Distillation for Pre-Training Small Foundation Models (CVPR 2024 · similarity 0.89)
- Exploring Target Representations for Masked Autoencoders (ICLR 2024 · similarity 0.89)
- From Prototypes to General Distributions: An Efficient Curriculum for Masked Image Modeling (CVPR 2025 · similarity 0.88)
- On the Role of Discrete Tokenization in Visual Representation Learning (ICLR 2024 · similarity 0.88)
- Self-Supervised Representation Learning from Arbitrary Scenarios (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.