Research map / Label-efficient and robust learning / Knowledge distillation
Asymmetric Decision-Making in Online Knowledge Distillation: Unifying Consensus and Divergence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Asymmetric Decision-Making (ADM), an online knowledge distillation method that reweights intermediate features by teacher-student spatial similarity (consensus loss for students, divergence loss for teachers) and adds a detached feature-MSE term, improving DML and other OKD baselines on CIFAR-100 and ImageNet.
Problem
Online knowledge distillation mostly uses logits-based mutual learning, leaving intermediate spatial feature similarity between peer teacher and student networks underexploited, and teachers can degrade under symmetric mimicry.
Approach
Use per-location cosine similarity between teacher and student intermediate features as a spatial weight: the student's classifier is trained on features weighted up where similarity is high (consensus), while the teacher's classifier is trained on features weighted up where similarity is low (divergence), plus an MSE feature loss where the teacher side is detached.
Key results
- On CIFAR-100 ResNet32x4-ResNet8x4 same-architecture pair, DML+ADM reaches 74.23% vs DML 73.95% and vanilla 72.33%, while SwitOKD+ADM reaches 74.63%.
- On ImageNet ResNet-34 teacher / ResNet-18 student, DML+ADM gives student top-1 71.59% vs DML 71.12%.
- On diffusion step distillation with 1 sampling step, PD+ADM improves FID from 15.926 to 13.939 on CIFAR-10.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ADM to DML | Online distillation with 0.5MobileNetV2 student and ResNet-18 teacher · CIFAR-100 | Top-1 accuracy (%): 66.69 -> 68.10 | Helped |
| Add ADM to DML | Online distillation with ResNet-18 student and ResNet-34 teacher · ImageNet | Top-1 accuracy (%): 71.12 -> 71.59 | Helped |
| Add ADM to DML | DeepLabV3 with ResNet-18 student · Cityscapes val | Score (metric unspecified): 74.19 -> 74.55 | Helped |
| Add ADM to PD | Diffusion step distillation · CIFAR-10, 1 sampling step | FID: 15.926 -> 13.939 | Helped |
| Add ADM to KD | Offline distillation with ResNet-18 student and ResNet-34 teacher · ImageNet validation | Top-1 accuracy (%): 71.03 -> 71.25 | Helped |
| Add feature distillation loss (ablation) | DML with ResNet-18 student and ResNet-34 teacher · ImageNet | Student accuracy: 71.12 -> 71.43 | Helped |
| Add consensus loss (ablation) | DML with ResNet-18 student and ResNet-34 teacher · ImageNet | Student accuracy: 71.12 -> 71.35 | Helped |
| Add divergence loss (ablation) | DML with ResNet-18 student and ResNet-34 teacher · ImageNet | Teacher accuracy: 73.78 -> 74.07 | Helped |
| Add feature distillation, consensus, and divergence losses together (ablation) | DML with ResNet-18 student and ResNet-34 teacher · ImageNet | Student accuracy: 71.12 -> 71.59 | Helped |
| Use KD for consensus loss while retaining CE for divergence loss (ablation) | ADM with ResNet-18 student and ResNet-34 teacher · ImageNet | Student accuracy: 71.41 -> 71.59 | Helped |
| Use KD for both consensus and divergence losses instead of KD/CE (ablation) | ADM with ResNet-18 student and ResNet-34 teacher · ImageNet | Student accuracy: 71.59 -> 71.45 | Hurt |
| Set gamma to 0.1 rather than 0.01 (ablation) | ADM with ResNet-18 student and ResNet-34 teacher · ImageNet | Student accuracy: 71.59 -> 71.48 | Hurt |
Surprising findings
- Teacher-side performance under existing OKD methods often degrades relative to vanilla training; ADM mitigates this in some settings.
- A larger ResNet-101 teacher fails to distill a better ResNet-18 student under several existing OKD methods, suggesting parameter-size gap is not the main cause.
Limitations
- Hyperparameters (alpha, beta, gamma) differ between CIFAR-100 and ImageNet, and tuning is needed across settings.
- Teacher performance can still decline in some OKD settings, and the authors attribute this to student-teacher gap constraints without fully resolving it.
- Diffusion experiments are limited (single loss applied for t<400 and a fixed teacher-student pair).
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet Cityscapes
Benchmarks: CIFAR-100 ImageNet Cityscapes CIFAR-10
Metrics: top-1 accuracy top-5 accuracy mIoU FID Inception Score
Baselines: DML KDCL SwitOKD SHAKE KD DKD DisWOT NORM AFD AFID PD
Builds on: Deep Mutual Learning FitNet KDCL SwitOKD SHAKE
Closest papers by meaning
- Less or More From Teacher: Exploiting Trilateral Geometry For Knowledge Distillation (ICLR 2024 · similarity 0.88)
- PLD: A Choice-Theoretic List-Wise Knowledge Distillation (NeurIPS 2025 · similarity 0.87)
- Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity (NeurIPS 2025 · similarity 0.87)
- Few-Shot Knowledge Distillation of LLMs With Counterfactual Explanations (NeurIPS 2025 · similarity 0.87)
- Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · similarity 0.87)
- ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via $\alpha$-$\beta$-Divergence (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.