Research map / Continual learning / Online and replay-based

Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspective

ICML 2025 · Continual learning › Online and replay-based · approach: Theoretical analysis of forgetting · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Dual-Arch, a plug-in continual learning framework that pairs a deep-thin plastic learner with a wide-shallow stable learner, transferring knowledge via distillation to balance stability and plasticity at the architectural level.

Problem

The stability-plasticity dilemma in continual learning is usually addressed at the parameter or learning-method level, while the architecture's role is overlooked; under equal parameter budgets, deeper and wider networks trade plasticity against stability.

Approach

Empirically show that deeper-narrower networks favor plasticity and wider-shallower networks favor stability under equal parameters, then train two independent networks with tailored architectures: a plastic learner trained with cross-entropy on current data, and a stable learner that distills from the frozen plastic teacher and combines with existing CL methods.

Key results

• Dual-Arch with DER on CIFAR100/10 reaches LA 66.22 and AIA 75.08 versus 61.83 and 72.48 for the original DER, using about 52% fewer parameters.
• Applied to WA on CIFAR100/10, Dual-Arch improves LA by 2.80 points (56.98 to 59.78) and reduces the parameter count by 33%.
• Dual-Arch reduces parameter counts by up to 87% while improving AIA by 0.90% (DER) and 1.94% (Foster) in the parameter-efficiency analysis.

Measured effects reported in the paper

Surprising findings

• Removing the plastic learner reduces AIA by an average of 2.63%, and using the same architecture for both learners also lowers performance, showing that architecture tailoring matters beyond simply having two models.
• Dual-Arch increases training time by 1.39x to 1.77x because the two learners cannot be trained in parallel, despite lower FLOPs at inference.

Limitations

• Training time increases by 1.39x to 1.77x because the two learners are trained sequentially and not in parallel.
• Evaluation focuses on ResNet-18 and a single ViT (SepViT); generality to other architectures is only partially validated.
• Architecture designs (Sta-Net, Pla-Net) are hand-crafted from a small empirical study rather than systematically searched.

Experimental setup

Datasets: CIFAR100 ImageNet100 GCIL-CIFAR-100 MNIST

Benchmarks: CIFAR100/10 CIFAR100/20 ImageNet100/10 ImageNet100/20 CIFAR100/50 GCIL-CIFAR-100

Metrics: last accuracy average incremental accuracy forgetting average accuracy on new tasks relative forgetting parameter count FLOPs training time

Baselines: iCaRL WA DER Foster MEMO ArchCraft

Builds on: iCaRL ResNet-18 PyCIL ArchCraft

Closest papers by meaning

• Achieving a Better Stability-Plasticity Trade-Off via Auxiliary Networks in Continual Learning (CVPR 2023 · similarity 0.93)
• NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks (ICML 2022 · similarity 0.90)
• Adapt Before Continual Learning (AAAI 2026 · similarity 0.90)
• Mitigating Plasticity Loss through Architectural Design in Continual Learning (ICML 2026 · similarity 0.90)
• New Insights for the Stability-Plasticity Dilemma in Online Continual Learning (ICLR 2023 · similarity 0.90)
• Recall-Oriented Continual Learning with Generative Adversarial Meta-Model (AAAI 2024 · similarity 0.88)