Research map / Label-efficient and robust learning / Knowledge distillation
Knowledge Distillation via the Target-Aware Transformer
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace one-to-one feature matching with target-aware one-to-all matching | student feature distillation · ImageNet validation | top-1 accuracy (%): 70.04 -> 72.07 | Helped |
| Replace one-to-one feature matching with target-aware one-to-all matching | student feature distillation · Pascal VOC validation | mIoU (%): 68.46 -> 73.85 | Helped |
| Replace one-to-one feature matching with target-aware one-to-all matching | student feature distillation · COCOStuff10k validation | mIoU (%): +2.42% | Helped |
| Add patch-group and anchor-point hierarchical distillation (ablation) | TaT semantic segmentation · Pascal VOC validation | mIoU (%): 72.07 -> 75.76 | Helped |
| Add anchor-point distillation (ablation) | TaT semantic segmentation · Pascal VOC validation | mIoU (%): 72.07 -> 75.37 | Helped |
| Add patch-group distillation (ablation) | TaT semantic segmentation · Pascal VOC validation | mIoU (%): 72.07 -> 75.63 | Helped |
| Use linear θ instead of identity θ (ablation) | TaT on ImageNet · ImageNet validation | top-1 accuracy (%): 72.22 -> 72.41 | Helped |
| Add a linear γ transformation as well as linear θ (ablation) | TaT on ImageNet · ImageNet validation | top-1 accuracy (%): 72.41 -> 72.35 | Hurt |
| Use 16 × 16 rather than 2 × 2 pooling kernels (ablation) | anchor-point distillation · Pascal VOC validation | mIoU (%): 75.37 -> 74.56 | Hurt |
| Reduce patch size from 32×32 to 8×8 (ablation) | patch-group distillation · Pascal VOC validation | mIoU (%): 75.33 -> 75.50 | Helped |
| Use 64 groups instead of 1 group (ablation) | patch-group distillation · Pascal VOC validation | mIoU (%): 75.26 -> 75.63 | Helped |
| Add LKL to TaT training | ResNet18 distilled by TaT · ImageNet validation | top-1 accuracy (%): 72.07 -> 72.41 | Helped |
Closest papers by meaning
- NORM: Knowledge Distillation via N-to-One Representation Matching (ICLR 2023 · similarity 0.88)
- Understanding the Role of the Projector in Knowledge Distillation (AAAI 2024 · similarity 0.88)
- FreeKD: Knowledge Distillation via Semantic Frequency Prompt (CVPR 2024 · similarity 0.87)
- DCSF-KD: Dynamic Channel-wise Spatial Feature Knowledge Distillation for Object Detection (AAAI 2025 · similarity 0.87)
- ScaleKD: Strong Vision Transformers Could Be Excellent Teachers (NeurIPS 2024 · similarity 0.87)
- Distilling Knowledge from Heterogeneous Architectures for Semantic Segmentation (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.