Research map / Continual learning / Online and replay-based
Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AID, a stochastic activation that applies interval-wise dropout to preactivations (simplified as negative/positive dropout masks), shown theoretically to regularize toward linear networks and empirically to mitigate plasticity loss in continual learning, warm-start, and Atari RL.
Problem
Neural networks lose plasticity (trainability and generalizability) under non-stationary or warm-start training, and standard Dropout does not mitigate this loss.
Approach
AID applies Dropout with different probabilities on different preactivation intervals, so it acts as a nonlinear stochastic activation; the simplified AID_p keeps positive values with probability p and negative with 1-p, which theoretically regularizes the network toward a deep linear network.
Key results
- On supervised generalization with learning rate decay, AID reached 0.690 on CIFAR100 (ResNet-18) versus 0.651 for Dropout and 0.581 for vanilla.
- In the warm-start experiment, the warm-start vs cold-start accuracy gap was 3.3 points for AID versus 10.9 for vanilla and 10.1 for Dropout.
- On 17 Atari games with replay ratio 1, AID significantly improved IQM human normalized score over vanilla DQN across 5 seeds.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the standard activation with AID | CNN · CIFAR10 | test accuracy: 0.742 -> 0.748 | Helped |
| Replace the standard activation with AID | ResNet-18 · CIFAR100 | test accuracy: 0.651 -> 0.690 | Helped |
| Replace the standard activation with AID | VGG-16 · TinyImageNet | test accuracy: 0.466 -> 0.519 | Helped |
| Use AID instead of the vanilla setting in warm-start training | ResNet-18 · CIFAR100 warm-start | warm-start/cold-start accuracy gap: 10.9\%p -> 3.3\%p | Helped |
| Use Dropout instead of the vanilla setting in warm-start training | ResNet-18 · CIFAR100 warm-start | warm-start/cold-start accuracy gap: 10.9\%p -> 10.1\%p | No clear effect |
| Use Dropout instead of vanilla training | neural networks in continual trainability tasks · Permuted MNIST and Random Label MNIST | training accuracy across tasks | Hurt |
Surprising findings
- Standard Dropout fails to mitigate plasticity loss and even degrades trainability in most conditions, including with random-label MNIST.
- Adding Dropout after ReLU provably does not yield the linearity-regularizing effect that AID has.
Limitations
- Experiments primarily use the simplified AID_p configuration; the general multi-interval AID is left underexplored.
- The link between the trainability explanation and the observed generalizability gains remains uncertain.
- The theoretical analysis is for a 2-layer network and the paper notes an error in a prior proof it corrected with a weaker inequality.
Experimental setup
Datasets: MNIST CIFAR10 CIFAR100 TinyImageNet Atari 2600 (17 games)
Benchmarks: Permuted MNIST Random Label MNIST CIFAR10 CIFAR100 TinyImageNet Atari 2600 (17 games)
Metrics: training accuracy test accuracy IQM human normalized score warm-start vs cold-start accuracy gap
Baselines: Dropout L2 regularization L2 Init Shrink & Perturb ReDo Continual Backprop CReLU RReLU Deep Fourier Features DropReLU DASH S-EWC
Builds on: Dropout DropReLU Randomized ReLU Shrink & Perturb Continual Backprop
Closest papers by meaning
- Activation Function Design Sustains Plasticity in Continual Learning (ICLR 2026 · similarity 0.87)
- Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churn (ICML 2025 · similarity 0.86)
- Stochastic Modified Equations and Dynamics of Dropout Algorithm (ICLR 2024 · similarity 0.85)
- Self-Normalized Resets for Plasticity in Continual Learning (ICLR 2025 · similarity 0.85)
- A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning (NeurIPS 2024 · similarity 0.85)
- Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stages (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.