Research map / Continual learning / Online and replay-based
Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adds auxiliary classifiers on intermediate layers of networks trained with continual learning methods, improving class-incremental accuracy by about 10% relative and enabling dynamic early-exit inference that cuts cost by 10-60%.
Problem
Catastrophic forgetting in class-incremental continual learning, plus the high inference cost of deep networks in resource-constrained settings.
Approach
Intermediate representations are more stable than final-layer ones during continual learning, so attaching auxiliary classifiers (ACs) to intermediate layers and combining their predictions (static inference) improves accuracy, while confidence-thresholded early exits (dynamic inference) reduce computation.
Key results
- Adding 6 ACs raises average final accuracy across methods from 34.60% to 39.15% on CIFAR100x5 (+4.55 points) and from 30.21% to 34.13% on CIFAR100x10.
- On ImageNet100x10, ACs raise average accuracy from 37.82% to 41.93%, with FT improving from 14.40% to 22.14%.
- AC-enhanced methods match single-classifier accuracy using about 50-70% of FLOPs on CIFAR100 and 40-60% on ImageNet100 under dynamic inference.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add auxiliary classifiers to the continual learning method | ResNet32 · CIFAR100x5 | average accuracy: 34.60 -> 39.15 | Helped |
| Add auxiliary classifiers to the continual learning method | ResNet32 · CIFAR100x10 | average accuracy: 30.21 -> 34.13 | Helped |
| Add auxiliary classifiers to the continual learning method | ResNet18 · ImageNet100x5 | average accuracy: 44.48 -> 50.01 | Helped |
| Add auxiliary classifiers to the continual learning method | ResNet18 · ImageNet100x10 | average accuracy: 37.82 -> 41.93 | Helped |
| Increase auxiliary classifier count from 6 to 12 (ablation) | ResNet32 · CIFAR100x5 | average accuracy improvement over baseline: +4.55 -> +4.74 | Helped |
| Increase auxiliary classifier count from 6 to 12 (ablation) | ResNet32 · CIFAR100x10 | average accuracy improvement over baseline: +4.20 -> +4.05 | Hurt |
| Use 18 rather than 6 auxiliary classifiers (ablation) | VGG19 · CIFAR100x5 | average accuracy: 42.61 -> 44.61 | Helped |
| Use 18 rather than 6 auxiliary classifiers (ablation) | VGG19 · CIFAR100x10 | average accuracy: 34.26 -> 36.79 | Helped |
| Replace independent auxiliary classifiers with cascading classifiers (ablation) | ResNet32 · CIFAR100x5 | average accuracy improvement over baseline: +2.73 -> +2.31 | Hurt |
| Replace independent auxiliary classifiers with cascading classifiers (ablation) | ResNet32 · CIFAR100x10 | average accuracy improvement over baseline: +4.20 -> +4.32 | Helped |
| Train auxiliary classifiers end-to-end rather than by linear probing (ablation) | ResNet32 · CIFAR100x5 | average accuracy: 38.09 -> 38.72 | Helped |
| Train auxiliary classifiers end-to-end rather than by linear probing (ablation) | ResNet32 · CIFAR100x10 | average accuracy: 33.06 -> 33.57 | Helped |
Surprising findings
- Intermediate classifiers trained on early layers can outperform the final classifier on older tasks, especially for exemplar-free methods such as FT and LwF.
- Overthinking (oracle accuracy minus final accuracy) is much more pronounced in continual learning than in i.i.d. joint training.
Limitations
- ACs add slight training and memory overhead that varies by method and architecture.
- Study limited to computer vision image classification.
- Exit threshold must be properly set to realize computational savings.
Experimental setup
Datasets: CIFAR100 ImageNet100
Benchmarks: CIFAR100 ImageNet100
Metrics: average final accuracy relative FLOPs for dynamic inference forgetting training time peak GPU memory
Baselines: FT FT+Ex GDumb ANCL BiC DER++ ER EWC LwF LODE SSIL
Builds on: early-exit networks FACIL BiC LwF DER++
Closest papers by meaning
- KAC: Kolmogorov-Arnold Classifier for Continual Learning (CVPR 2025 · similarity 0.89)
- Learning to Continually Learn with the Bayesian Principle (ICML 2024 · similarity 0.89)
- Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learning (ICLR 2024 · similarity 0.89)
- Convergence and Implicit Bias of Gradient Descent on Continual Linear Classification (ICLR 2025 · similarity 0.88)
- Continual learning with the neural tangent ensemble (NeurIPS 2024 · similarity 0.88)
- RanDumb: Random Representations Outperform Online Continually Learned Representations (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.