Research map / Label-efficient and robust learning / Knowledge distillation
Structured Knowledge Distillation for Dense Prediction
Yifan Liu, Changyong Shu, Jingdong Wang, Chunhua Shen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add pixel-wise distillation (ablation) | ResNet18 (0.5) · Cityscapes val | mIoU: 55.37 -> 57.07 | Improved |
| add pair-wise distillation (ablation) | ResNet18 (0.5) + PI · Cityscapes val | mIoU: 57.07 -> 61.52 | Improved |
| add holistic distillation (ablation) | ResNet18 (0.5) + PI + PA · Cityscapes val | mIoU: 61.52 -> 62.35 | Improved |
| add pair-wise distillation (ablation) | ResNet18 (1.0) ImageNet + PI · Cityscapes val | mIoU: 70.51 -> 71.78 | Improved |
| add holistic distillation (ablation) | ResNet18 (1.0) ImageNet + PI + PA · Cityscapes val | mIoU: 71.78 -> 74.08 | Improved |
| replace MIMIC feature distillation by pair-wise distillation (ablation) | ResNet18 (1.0) ImageNet + PI · Cityscapes val | mIoU: 71.03 -> 71.78 | Improved |
| replace local pair-wise distillation by fully connected pair-wise (ablation) | ResNet18 (1.0) ImageNet + PI · Cityscapes val | mIoU: 70.86 -> 71.78 | Improved |
| use fully connected affinity graph instead of short connection range (ablation) | pair-wise distillation · Cityscapes val | mIoU: 70.83 -> 71.37 | Improved |
| use 2x2 node patches instead of single-pixel nodes (ablation) | pair-wise distillation · Cityscapes val | mIoU: 71.37 -> 71.78 | Improved |
| fuse pair-wise distillation across feature levels (ablation) | pair-wise distillation · Cityscapes val | mIoU: 71.78 -> 72.18 | Improved |
| add self-attention layers to the discriminator (A2L4 vs A0L4) (ablation) | holistic distillation · Cityscapes val | mIoU: 72.85 -> 74.08 | Improved |
| use deeper discriminator (4 conv blocks instead of 2) (ablation) | holistic distillation · Cityscapes val | mIoU: 72.24 -> 74.08 | Improved |
Closest papers by meaning
- Structured Knowledge Distillation for Semantic Segmentation (CVPR 2019 · similarity 0.94)
- Channel-Wise Knowledge Distillation for Dense Prediction (ICCV 2021 · similarity 0.91)
- Multi-Task Learning with Knowledge Distillation for Dense Prediction (ICCV 2023 · similarity 0.89)
- Structured Knowledge Distillation for Accurate and Efficient Object Detection (TPAMI 2023 · similarity 0.88)
- Prediction-Guided Distillation for Dense Object Detection (ECCV 2022 · similarity 0.88)
- Distilling Knowledge From Graph Convolutional Networks (CVPR 2020 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
