Research map / Continual learning / Class-incremental
Gradient Reweighting: Towards Imbalanced Class-Incremental Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes gradient reweighting plus a distribution-aware knowledge distillation loss to tackle intra- and inter-phase class imbalance in class-incremental learning trained from scratch end-to-end.
Problem
Class-incremental learning on long-tailed data suffers from a dual imbalance (intra-phase class imbalance within tasks and inter-phase imbalance between exemplars and new data), causing biased FC gradients, biased weights, and imbalanced forgetting.
Approach
Reweight FC-layer gradients per class using accumulated gradient magnitudes (class-balance ratio), decouple reweighting between old and new classes with a task-balance ratio and loss-balance ratio, and use a distribution-aware KD loss that aligns logits with the distribution of lost training data.
Key results
- On ImageNetSubset-LT under LFS with N=10 tasks, the method reaches 45.12% average accuracy versus 40.01% for MAFDRC (best prior in the table at that setting).
- Under LFH with N=10 on CIFAR100-LT, the method achieves 34.35% vs 32.67% for MAFDRC at N=10 LFS setting comparisons in Table 1 (CIFAR100-LT LFS N=10: 35.66%).
- Ablation on Food101-LT shows gradient reweighting plus DAKD and DGR together raise LFS N=10 accuracy from 27.77% to 29.05%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use gradient reweighting and DAKD instead of the baseline GR + KD configuration (ablation) | Gradient Reweighting · Food101-LT, LFS N=10 | average accuracy (%): 27.77 -> 29.05 | Helped |
| Use gradient reweighting and DAKD instead of the baseline GR + KD configuration (ablation) | Gradient Reweighting · ImageNetSubset-LT, LFS N=10 | average accuracy (%): 47.42 -> 50.57 | Helped |
| Add DAKD to GR + KD (ablation) | Gradient Reweighting · Food101-LT, LFS N=10 | average accuracy (%): 27.77 -> 28.45 | Helped |
| Add DAKD to GR + KD (ablation) | Gradient Reweighting · ImageNetSubset-LT, LFS N=10 | average accuracy (%): 47.42 -> 48.64 | Helped |
| Use DGR instead of GR to separately balance new and learned task gradients (ablation) | Gradient Reweighting · Food101-LT, LFS N=10 | average accuracy (%): 28.02 vs 28.96 | Hurt |
| Use DGR instead of GR to separately balance new and learned task gradients (ablation) | Gradient Reweighting · ImageNetSubset-LT, LFS N=10 | average accuracy (%): 48.72 vs 49.59 | Helped |
| Preserve the discriminative gradients from DAKD rather than reweighting them (ablation) | Gradient Reweighting · Food101-LT, LFS N=10 | average accuracy (%): 28.96 -> 29.05 | Helped |
| Preserve the discriminative gradients from DAKD rather than reweighting them (ablation) | Gradient Reweighting · ImageNetSubset-LT, LFS N=10 | average accuracy (%): 49.59 -> 50.57 | Helped |
| Use DAKD instead of balanced KD, multi-level KD, decoupled KD, or decomposed KD (ablation) | Gradient Reweighting · ImageNetSubset-LT, LFS N=10 | forgetting | Helped |
| Increase the memory budget from nε=10 to nε=50 | CIL methods · ImageNetSubset-LT and Food101-LT | accuracy (%) | Mixed |
| Increase the imbalance factor ρ from 50 to 150 | CIL methods · Food101-LT | accuracy (%) | Hurt |
Surprising findings
- Imbalanced forgetting: instance-rich (head) classes suffer higher performance degradation in CIL, contrary to prior findings.
- Increasing memory budget can yield marginal gains for some methods because exemplar sets become more class-imbalanced.
Limitations
- Hyperparameters set to fixed values (gamma=1, lambda_b=1) for simplicity, without tuning analysis.
- Evaluated only on three image datasets with ResNet backbones trained from scratch.
Experimental setup
Datasets: CIFAR100-LT ImageNetSubset-LT Food101-LT
Benchmarks: CIFAR100-LT ImageNetSubset-LT Food101-LT
Metrics: average accuracy forgetting
Baselines: iCaRL IL2M BiC WA SSIL FOSTER MAFDRC EEIL-2stage LUCIR-2stage PODNet-2stage FOSTER-2stage
Builds on: iCaRL logit adjustment (regularized softmax) Decoupled Knowledge Distillation Learning a Unified Classifier Incrementally (LUCIR) Weight Aligning (WA)
Closest papers by meaning
- Defying Imbalanced Forgetting in Class Incremental Learning (AAAI 2024 · similarity 0.91)
- Long-Tail Class Incremental Learning via Independent Sub-prototype Construction (CVPR 2024 · similarity 0.90)
- Rebalancing Multi-Label Class-Incremental Learning (AAAI 2025 · similarity 0.90)
- Addressing Imbalanced Domain-Incremental Learning through Dual-Balance Collaborative Experts (ICML 2025 · similarity 0.89)
- eTag: Class-Incremental Learning via Embedding Distillation and Task-Oriented Generation (AAAI 2024 · similarity 0.88)
- Class Incremental Learning with Multi-Teacher Distillation (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.