Research map / Continual learning / Class-incremental

Maintaining Fairness in Logit-based Knowledge Distillation for Class-Incremental Learning

AAAI 2025 · Continual learning › Class-incremental · approach: Class-balanced knowledge retention · 8 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a plug-and-play Z-score logit normalization (inter-class) plus intra-class KD loss to resolve the conflict between logit-based KD and cross-entropy in class-incremental learning, improving recency bias and FAA/CAA of several KD-based CIL methods.

Problem

Logit-based knowledge distillation in class-incremental learning enforces exact logit matching for old classes, which conflicts with cross-entropy learning of new classes, causing recency bias and unfairness between old and new classes (and within old classes).

Approach

Z-score normalize student and teacher logits across all classes (not only old ones) before distillation, so the KD loss matches semantic inter-class relations rather than exact values; additionally add an intra-class KD loss across instances in a batch to address overconfident teacher predictions and fairness within old classes.

Key results

• Applying the inter-class loss plus intra-class loss to LwF on CIFAR-100 Half 6 Tasks improves FAA by 5.81 and CAA by 5.59 on average across methods.
• Average improvements across methods are up to 5.81% FAA and 5.59% CAA on CIFAR-100, and consistent gains on TinyImageNet (e.g., LwF Split 5 Tasks FAA 23.05 to 28.20).
• The method adds no additional training cost and integrates with LwF, Replay, iCaRL, BiC, WA and PODNet.

Measured effects reported in the paper

Surprising findings

• Modifying the student's softmax to include new-class logits in the KD term substantially reduces temporary forgetting, yet the new-task accuracy stays at 0 because the exact match makes the student mirror the teacher's predictions.

Limitations

• Gains vary across settings; some configurations (e.g., BiC Half 11 Tasks intra-loss) show slight drops.
• Hyperparameter sensitivity: performance depends on the inter/intra loss weight ratio and batch size for intra-class distillation.
• Evaluated only with ResNet-18 on image classification benchmarks.

Experimental setup

Datasets: CIFAR-100 ImageNet-Subset TinyImageNet

Benchmarks: CIFAR-100 ImageNet-Subset TinyImageNet

Metrics: Final Average Accuracy (FAA) Cumulative Average Accuracy (CAA)

Baselines: LwF Replay iCaRL BiC WA PODNet

Builds on: LwF iCaRL BiC WA PODNet

Closest papers by meaning

• Can Students Beyond the Teacher? Distilling Knowledge from Teacher’s Bias (AAAI 2025 · similarity 0.90)
• PLD: A Choice-Theoretic List-Wise Knowledge Distillation (NeurIPS 2025 · similarity 0.90)
• Logit Standardization in Knowledge Distillation (CVPR 2024 · similarity 0.89)
• Scaled Decoupled Distillation (CVPR 2024 · similarity 0.89)
• Enhancing Logits Distillation with Plug&Play Kendall's $\tau$ Ranking Loss (ICML 2025 · similarity 0.89)
• Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models (NeurIPS 2026 · similarity 0.88)