Research map / Label-efficient and robust learning / Knowledge distillation
Self-Regulated Feature Learning via Teacher-Free Feature Distillation
Lujun Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add all intra-layer Tf-FD losses on top of nothing (ablation) | ResNet-20 · CIFAR-100 | top-1 accuracy (%): 69.06 -> 70.51 | Improved |
| add intra-layer Tf-FD at the deepest stage S4 rather than S2 (ablation) | ResNet-20 · CIFAR-100 | top-1 accuracy (%): 70.26 -> 70.41 | Improved |
| add all three inter-layer losses (ablation) | ResNet-20 · CIFAR-100 | top-1 accuracy (%): 69.06 -> 70.31 | Improved |
| add intra-layer loss to inter-layer losses (full Tf-FD) (ablation) | ResNet-20 · CIFAR-100 | top-1 accuracy (%): 70.31 -> 70.62 | Improved |
| add a single Tf-FD loss (ablation) | ResNet-20 · CIFAR-100 | top-1 accuracy (%): 0.78 ~ 1.23 gains | Improved |
| add logits KD to Tf-FD (Tf-FD†) | ResNet-20 · CIFAR-100 | top-1 accuracy (%): 70.62 -> 71.56 | Improved |
| replace DropBlock feature regularization with Tf-FD | ResNet-20 · CIFAR-100 | top-1 accuracy (%): 69.65 -> 70.62 | Improved |
| add Tf-FD to ResNet-18 | ResNet-18 · ImageNet val | top-1 accuracy (%): 69.75 -> 70.46 | Improved |
| add Tf-FD† to ResNet-18 | ResNet-18 · ImageNet val | top-1 accuracy (%): 69.75 -> 71.00 | Improved |
| add Tf-FD† to Faster R-CNN R50-FPN | Faster R-CNN R50-FPN · MS-COCO2017 val | AP: 37.93 -> 38.92 | Improved |
| replace l2 mimicking loss with AT, SP or ICKD (ablation) | Tf-FD ResNet-20 · CIFAR-100 | top-1 accuracy (%) | Improved |
| replace l2 mimicking loss with AB (ablation) | Tf-FD ResNet-20 · CIFAR-100 | top-1 accuracy (%) | Hurt |
Closest papers by meaning
- Task-Oriented Feature Distillation (NeurIPS 2020 · similarity 0.85)
- A Comprehensive Overhaul of Feature Distillation (ICCV 2019 · similarity 0.84)
- SDPGO: Efficient Self-Distillation Training Meets Proximal Gradient Optimization (NeurIPS 2025 · similarity 0.84)
- Class Incremental Learning with Multi-Teacher Distillation (CVPR 2024 · similarity 0.81)
- M2SD:Multiple Mixing Self-Distillation for Few-Shot Class-Incremental Learning (AAAI 2024 · similarity 0.81)
- Rethinking Self-Distillation: Label Averaging and Enhanced Soft Label Refinement with Partial Labels (ICLR 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
