Research map / Label-efficient and robust learning / Knowledge distillation
HAD: Heterogeneity-Aware Distillation for Lifelong Heterogeneous Learning
Xuerui Zhang, Xuehao Wang, Zhan Zhuang, Linglan Zhao, Ziyue Li, Xinmin Zhang, Zhihuan Song, Yu Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add distribution-balanced loss (DB-HAD) (ablation) | HAD · NYUv2 | avg relative improvement: -2.84% when removed | Improved |
| add salience-guided loss (SG-HAD) (ablation) | HAD · NYUv2 | avg relative improvement: -1.27% when removed | Improved |
| use geometric instead of arithmetic mean within groups (ablation) | HAD · NYUv2 | avg relative improvement: -6.29% with arithmetic | Improved |
| use binary foreground/background grouping instead of equal-interval groups for regression (ablation) | HAD · NYUv2 | avg relative improvement: -3.64% (5), -5.43% (10), -9.72% (15) groups | Improved |
| distill into full network (DIS) instead of updating only decoders or only new decoder (ablation) | HAD · NYUv2 | avg relative improvement: -7.76%, -6.65%, -10.94% for alternatives | Improved |
| HAD vs EWC on NYUv2 ResNet-18 | lifelong learner · NYUv2 | avg relative improvement: +24.39% -> +32.74% | Improved |
| use larger encoder ResNet-50 with HAD | HAD · NYUv2 | avg relative improvement: +32.74% -> +35.71% | Improved |
| sequential HAD vs separate task-specific models | HAD · NYUv2 | per-task improvement: +1.38% seg, +5.2% depth, +9.87% normal | Improved |
Closest papers by meaning
- Heterogeneity-aware Distillation for Federated Continual Learning (NeurIPS 2026 · similarity 0.88)
- Multi-Task Learning with Knowledge Distillation for Dense Prediction (ICCV 2023 · similarity 0.87)
- Task Groupings Regularization: Data-Free Meta-Learning with Heterogeneous Pre-trained Models (ICML 2024 · similarity 0.85)
- Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity (NeurIPS 2025 · similarity 0.85)
- ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via $\alpha$-$\beta$-Divergence (ICML 2025 · similarity 0.84)
- Progressively Knowledge Distillation via Re-parameterizing Diffusion Reverse Process (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
