Research map / Label-efficient and robust learning / Self-supervised and contrastive
Supervised Masked Knowledge Distillation for Few-Shot Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add self-supervised MIM pretraining before supervised SMKD training (ablation) | SMKD on mini-ImageNet · mini-ImageNet | accuracy: 33.14±0.13 -> 74.28±0.18; 45.02±0.14 -> 88.82±0.09 | Helped |
| Combine class-level and patch-level distillation losses (ablation) | SMKD · mini-ImageNet | accuracy: 70.21±0.17 -> 74.28±0.18; 87.03±0.10 -> 88.82±0.09 | Helped |
| Add class-level distillation alongside patch-level distillation (ablation) | SMKD · mini-ImageNet | accuracy: 70.84±0.18 -> 74.28±0.18; 85.90±0.11 -> 88.82±0.09 | Helped |
| Replace class-level distillation with cross-entropy while retaining patch loss (ablation) | SMKD · mini-ImageNet | accuracy: 70.70±0.18 -> 74.28±0.18; 86.77±0.10 -> 88.82±0.09 | Hurt |
| Add MIM to class-token self-supervision in stage 1 (ablation) | SMKD pretraining · mini-ImageNet | accuracy: 58.55 -> 60.93; 78.90 -> 80.38 | Helped |
| Use blockwise masking instead of no masking during pretraining (ablation) | SMKD pretraining · mini-ImageNet | accuracy: 59.15±0.17 -> 60.93±0.17; 79.23±0.12 -> 80.38±0.12 | Mixed |
| Use blockwise masking instead of random masking (ablation) | SMKD pretraining and supervised training · mini-ImageNet | accuracy: 60.94±0.18 -> 60.93±0.17; 79.62±0.13 -> 80.38±0.12; 74.07±0.18 -> 74.01±0.17; 88.66±0.09 -> 88.89±0.09 | Mixed |
| Replace simple average patch-loss weighting with self-attention weighting (ablation) | SMKD patch distillation · mini-ImageNet | accuracy: 74.28±0.18 -> 74.11±0.18; 88.82±0.09 -> 88.52±0.10 | Hurt |
| Concatenate cls features with weighted-average patch features at evaluation (ablation) | SMKD few-shot evaluation · mini-ImageNet | accuracy: 71.26±0.17 -> 74.28±0.18; 88.03±0.10 -> 88.82±0.09 | Helped |
| Add patch-size-8 processing to SMKD (ablation) | SMKD with Prototype/Classifier evaluation · mini-ImageNet | accuracy: 74.28±0.18 -> 74.00±0.17; 88.82±0.09 -> 89.57±0.09 | Mixed |
| Add spectral token pooling to SMKD with patch size 8 (ablation) | SMKD · mini-ImageNet | accuracy: 74.00±0.17 -> 75.32±0.18; 89.57±0.09 -> 89.23±0.18 | Mixed |
| Allow distillation between intra-class global and local crops (ablation) | SMKD self-supervised pretraining · few-shot classification | accuracy | Hurt |
Closest papers by meaning
- Asymmetric Masked Distillation for Pre-Training Small Foundation Models (CVPR 2024 · similarity 0.89)
- Pushing the Limits of Simple Pipelines for Few-Shot Learning: External Data and Fine-Tuning Make a Difference (CVPR 2022 · similarity 0.89)
- A Theoretical Analysis of Self-Supervised Learning for Vision Transformers (ICLR 2025 · similarity 0.88)
- Vision Transformers Need Registers (ICLR 2024 · similarity 0.88)
- Frequency-Guided Masking for Enhanced Vision Self-Supervised Learning (ICLR 2025 · similarity 0.88)
- One Meta-tuned Transformer is What You Need for Few-shot Learning (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.