Research map / Label-efficient and robust learning / Knowledge distillation
Distilling Knowledge from Heterogeneous Architectures for Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HeteroAKD, a knowledge distillation method for semantic segmentation that transfers knowledge across heterogeneous CNN and Transformer teacher-student pairs via logits-space projection, a label-guided knowledge mixing mechanism, and a knowledge evaluation mechanism that reweights pixels.
Problem
Existing knowledge distillation methods for semantic segmentation assume homogeneous teacher and student architectures; distilling across heterogeneous architectures (CNN and Transformer with different inductive biases) is challenging due to feature disparities and teacher predictions not always bei
Approach
Project intermediate features of both teacher and student into an aligned logits space to remove architecture-specific information, then use label-based cross-entropy reliability to mix teacher and student knowledge (KMM) and reweight distillation pixels by the reliability discrepancy (KEM).
Key results
- On Cityscapes, HeteroAKD raises SegFormer-MiT-B1 student from 74.91 to 76.34 mIoU with a DeepLabV3-Res101 teacher, versus 75.20 for the best prior method (Af-DCD).
- On ADE20K with DeepLabV3-Res101 teacher, HeteroAKD reaches 38.84 mIoU for SegFormer-MiT-B1 vs 35.18 baseline and 36.74 for Af-DCD.
- Ablation on Cityscapes shows removing KMM costs 0.35% and removing KEM costs 0.49% average mIoU; combined losses give 1.66% average gain.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply HeteroAKD instead of the strongest listed KD baseline | heterogeneous teacher-student segmentation pairs · Cityscapes, Pascal VOC, ADE20K validation | mIoU | Helped |
| Add Lkd to the baseline (ablation) | heterogeneous teacher-student pairs · Cityscapes val, both distillation modes | mIoU: 0.94% average mIoU gain | Helped |
| Add Lhakd to the baseline (ablation) | heterogeneous teacher-student pairs · Cityscapes val, both distillation modes | mIoU: 1.08% average mIoU gain | Helped |
| Add both Lkd and Lhakd (ablation) | heterogeneous teacher-student pairs · Cityscapes val, both distillation modes | mIoU: 1.66% average mIoU gain | Helped |
| Add KMM (ablation) | HeteroAKD · Cityscapes val, both distillation modes | mIoU: 0.35% average mIoU reduction when removed | Helped |
| Add KEM (ablation) | HeteroAKD · Cityscapes val, both distillation modes | mIoU: 0.49% average mIoU reduction when removed | Helped |
| Use HeteroAKD with a homogeneous teacher rather than a heterogeneous teacher | DeepLabV3-Res18 student · Cityscapes | mIoU gain: +0.62% vs +0.48% | Mixed |
Surprising findings
- In some cases the student surpasses the teacher after distillation (e.g., DeepLabV3-MiT-B4 to DeepLabV3-Res18 student exceeds teacher by 0.46%).
- Some existing KD methods (e.g., SKD, CWD) degrade performance below the student baseline in heterogeneous settings.
Limitations
- Heterogeneous teachers can yield lower distillation efficiency than homogeneous teachers in some cases, as admitted in the conclusion.
- Requires warming up the student under full label supervision before distillation.
- Hyperparameter optima (temperature, lambda weights) differ between the two transfer modes.
Experimental setup
Datasets: Cityscapes Pascal VOC ADE20K
Benchmarks: Cityscapes Pascal VOC ADE20K
Metrics: mIoU Params FLOPs
Baselines: SKD IFVD CWD CIRKD Af-DCD C2VKD
Builds on: CIRKD Af-DCD CWD SKD
Closest papers by meaning
- Heterogeneous Complementary Distillation (AAAI 2026 · similarity 0.91)
- One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillation (NeurIPS 2023 · similarity 0.90)
- Multi-Knowledge Aggregation and Transfer for Semantic Segmentation (AAAI 2022 · similarity 0.90)
- Generalizable Knowledge Distillation from Vision Foundation Models for Semantic Segmentation (CVPR 2026 · similarity 0.90)
- Structured Knowledge Distillation for Semantic Segmentation (CVPR 2019 · similarity 0.90)
- Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.