Research map / Label-efficient and robust learning / Knowledge distillation
Asymmetric Masked Distillation for Pre-Training Small Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Asymmetric Masked Distillation (AMD), where a teacher MAE sees more patches (lower masking ratio) than the student and serial direct and generation feature alignment distills small ViT models during masked autoencoding pre-training.
Problem
Masked autoencoding pre-training yields strong models mainly at large scale, which are costly to fine-tune; prior MAE distillation with symmetric masking limits the teacher's context and retains high masking-induced information loss.
Approach
Use asymmetric masking so the teacher sees a superset of the student's visible patches (lower teacher masking ratio, student keeps default high ratio), and align features serially: direct alignment on shared visible tokens, then generation alignment where a small decoder-like generator predicts teacher features for the extra visible tokens.
Key results
- AMD ViT-B pre-trained 800 epochs reaches 73.3% top-1 on SSV2, 3.7% above VideoMAE-B (69.6%) under the same schedule.
- AMD with MAE-L teacher reaches 84.6% top-1 on ImageNet-1K with ViT-B, surpassing ImageMAE by 1.0%.
- AMD ViT-B transfer from K400 improves SSV2 by 4.1%, UCF101 by 1.0% and HMDB51 by 6.3% over VideoMAE-B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AMD instead of VideoMAE pre-training | ViT-B · SSV2 | Top-1 accuracy: 69.6 -> 73.3 | Helped |
| use AMD instead of VideoMAE pre-training | ViT-B · K400 | Top-1 accuracy: 80.0 -> 82.2 | Helped |
| use AMD instead of ImageMAE pre-training | ViT-B · ImageNet-1K | Top-1 accuracy: 83.6 -> 84.6 | Helped |
| transfer AMD pre-training instead of VideoMAE pre-training | ViT-B · K400 → SSV2 | Top-1 accuracy: 68.5 -> 72.6 | Helped |
| transfer AMD pre-training instead of VideoMAE pre-training | ViT-B · K400 → UCF101 | Top-1 accuracy: 96.1 -> 97.1 | Helped |
| transfer AMD pre-training instead of VideoMAE pre-training | ViT-B · K400 → HMDB51 | Top-1 accuracy: 73.3 -> 79.6 | Helped |
| add intermediate K400 fine-tuning with labels | AMD ViT-B · AVA v2.2 | mAP: 29.9 -> 33.5 | Helped |
| increase generator depth from 1 to 2 (ablation) | AMD ViT-B · SSV2 | Top-1 accuracy: 72.06 -> 72.38 | Helped |
| increase generator depth from 2 to 4 (ablation) | AMD ViT-B · SSV2 | Top-1 accuracy: 72.38 -> 72.45 | Helped |
| align two layers instead of one (ablation) | AMD ViT-B · SSV2 | Top-1 accuracy: 71.93 -> 72.38 | Helped |
| lower teacher masking ratio from 85% to 45% (ablation) | AMD ViT-B · SSV2 | Top-1 accuracy: 72.17 -> 72.48 | Helped |
| combine direct and generation alignment serially instead of in parallel (ablation) | AMD ViT-B · SSV2 | Top-1 accuracy: 72.25 -> 72.38 | Helped |
Surprising findings
- Setting the teacher masking ratio to 0% (no teacher masking) gave worse performance (72.1%) than a 45% teacher masking ratio (72.5%), since the generation alignment becomes harder.
- Plain feature distillation without masking is more costly yet only matches the asymmetric variant's accuracy (72.4% vs 72.5%).
Limitations
- Teacher masking ratio is constrained by the computational cost of the teacher, so the asymmetry cannot be pushed far.
- Gains are evaluated mainly at 800 epochs or 200-epoch ablations; the method adds a teacher forward pass and generator overhead.
Experimental setup
Datasets: Something-Something V2 Kinetics-400 ImageNet-1K UCF101 HMDB51 AVA
Benchmarks: Something-Something V2 Kinetics-400 ImageNet-1K UCF101 HMDB51 AVA
Metrics: top-1 accuracy top-5 accuracy mAP
Baselines: VideoMAE DMAE ImageMAE G2SD SSTA MAE-ST MViT Motionformer
Builds on: VideoMAE ImageMAE DMAE knowledge distillation
Closest papers by meaning
- Exploring Target Representations for Masked Autoencoders (ICLR 2024 · similarity 0.90)
- Hybrid Distillation: Connecting Masked Autoencoders with Contrastive Learners (ICLR 2024 · similarity 0.89)
- Understanding and Enhancing Mask-Based Pretraining towards Universal Representations (NeurIPS 2025 · similarity 0.88)
- Closest Neighbors are Harmful for Lightweight Masked Auto-encoders (CVPR 2025 · similarity 0.87)
- Dataset Distillation for Pre-Trained Self-Supervised Vision Models (NeurIPS 2025 · similarity 0.87)
- VideoMAC: Video Masked Autoencoders Meet ConvNets (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.