Research map / Label-efficient and robust learning / Knowledge distillation
Debiased Distillation for Consistency Regularization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Intra-class Knowledge Distillation (IKD), a plug-and-play regularizer that distills teacher logits averaged per class (fixed after the first epoch) into the student to enforce intra-class prediction consistency.
Problem
Logit-based knowledge distillation with high temperature can lower target-class confidence, and intra-batch distillation methods are affected by random sampling, batch size, and noise, leading to biased predictions.
Approach
Store per-class average teacher logits in a C×C matrix computed once in the first epoch, then add a KL loss between the student output and the class-averaged teacher distribution indexed by label, so all samples of a class share a stable target; this is added on top of existing logit KD methods (KD, DKD, NKD, WTTM).
Key results
- On CIFAR-100, KD+IKD reaches 72.00% on ResNet56→ResNet20-style homogeneous setups, with average improvements of 0.63% to 1.44% over plug-and-play baselines (DOT, CTKD, SDD, LSKD).
- On ImageNet-1K with ResNet34→ResNet18, KD+IKD improves top-1 by +1.16 over KD (71.82 vs 70.66 in the homogeneous table), with Ψ average gain of +0.57 top-1.
- Table 1 shows IKD keeps CIFAR-100 accuracy between 76.04 and 76.75 across batch sizes 32-256, versus DIST's max gap of 2.14 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add IKD to KD | KD with ResNet32×4 teacher and ResNet8×4 student · CIFAR-100 validation | accuracy: 73.33 -> 76.56 | Helped |
| Add IKD to KD | KD with ResNet34 teacher and ResNet18 student · ImageNet-1K validation | Top-1 accuracy: 70.66 -> 71.82 | Helped |
| Add IKD to KD | KD with ResNet34 teacher and ResNet18 student · ImageNet-1K validation | Top-5 accuracy: 89.88 -> 90.57 | Helped |
| Add IKD to KD | KD with ResNet50 teacher and MobileNetV1 student · ImageNet-1K validation | Top-1 accuracy: 70.49 -> 73.27 | Helped |
| Add IKD to KD | KD with ResNet50 teacher and MobileNetV1 student · ImageNet-1K validation | Top-5 accuracy: 89.92 -> 91.34 | Helped |
| Use IKD rather than DIST across batch sizes 32, 64, 128, and 256 | CIFAR-100 classification with ResNet32×4 teacher and ResNet8×4 student · CIFAR-100 test | accuracy: 76.47 -> 76.75; 76.04 -> 76.56; 75.27 -> 76.13; 74.33 -> 76.04 | Helped |
| Lower the intra-class temperature T (ablation) | IKD · CIFAR-100 | classification accuracy | Helped |
| Set intra-class temperature T to 1.0 (ablation) | IKD · CIFAR-100 | classification accuracy | Helped |
| Vary λavg; use 6.5 for DKD+IKD (ablation) | DKD+IKD · CIFAR-100 | classification accuracy | Mixed |
| Vary λavg; use 6.0 for NKD+IKD (ablation) | NKD+IKD · CIFAR-100 | classification accuracy | Mixed |
Surprising findings
- NKD+IKD yields lower top-5 accuracy than NKD on ImageNet-1K homogeneous setting, which the authors attribute to NKD emphasizing non-target classes conflicting with IKD's focus on target classes.
Limitations
- Only logit-layer distillation is considered; feature-level knowledge is not captured.
- Validated only on image classification; generalization to other tasks such as person re-ID is untested.
- Hyperparameters (intra-class temperature T, weight λavg) need tuning and performance fluctuates with λavg.
Experimental setup
Datasets: CIFAR-100 ImageNet-1K Tiny-ImageNet
Benchmarks: CIFAR-100 ImageNet-1K Tiny-ImageNet
Metrics: top-1 accuracy top-5 accuracy
Baselines: KD DKD NKD WTTM CTKD DOT LSKD SDD DIST MLKD
Builds on: knowledge distillation DKD NKD WTTM DIST
Closest papers by meaning
- Self-Distillation From the Last Mini-Batch for Consistency Regularization (CVPR 2022 · similarity 0.91)
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.88)
- Logit Standardization in Knowledge Distillation (CVPR 2024 · similarity 0.88)
- Enhancing Class-Imbalanced Learning with Pre-Trained Guidance through Class-Conditional Knowledge Distillation (ICML 2024 · similarity 0.88)
- Scaled Decoupled Distillation (CVPR 2024 · similarity 0.88)
- Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge Distillation (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.