Research map / Continual learning / Online and replay-based
Plastic Learning with Deep Fourier Features
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows linear learners and deep diagonal linear nets avoid loss of plasticity, then proposes deep Fourier features (sin and cos in every layer) as a drop-in ReLU replacement that sustains trainability and improves continual learning on CIFAR10/100 and tiny-ImageNet.
Problem
Deep neural networks lose plasticity (trainability) when learning from non-stationary data streams; it is unclear what inductive biases sustain learning continually.
Approach
Concatenate sin and cos of every pre-activation in every layer so that one of each pair is approximately linear (within a bounded error), letting the network approximately embed a deep linear network while remaining nonlinear; the sign-entropy of units is used to explain why ReLU and other activations lose plasticity.
Key results
- Deep Fourier features are among the highest test-accuracy methods on diminishing label noise across ten tasks, with the largest gains on tiny-ImageNet.
- In class-incremental learning, deep Fourier features largely outperform baselines on tiny-ImageNet, while on CIFAR100 they remain among the top methods.
- Deep Fourier features network has roughly half the parameters of the ReLU ResNet-18 yet surpasses baselines on CIFAR100 and matches spectral regularization on CIFAR10.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace ReLU activations with deep Fourier features | ResNet-18 · Diminishing label noise on tiny-ImageNet | test accuracy | Helped |
| Replace ReLU activations with deep Fourier features | ResNet-18 · Diminishing label noise on CIFAR100 | test accuracy | Helped |
| Replace ReLU activations with deep Fourier features | ResNet-18 · Diminishing label noise on CIFAR10 | test accuracy | Mixed |
| Replace ReLU activations with deep Fourier features | ResNet-18 · Class-incremental learning on tiny-ImageNet | test accuracy | Helped |
| Replace ReLU activations with deep Fourier features | ResNet-18 · Class-incremental learning on CIFAR100 | test accuracy | Mixed |
| Replace ReLU activations with deep Fourier features | ResNet-18 · Continual learning across the considered datasets | test accuracy | Helped |
| Add spectral regularization to deep Fourier features | Deep Fourier features · Continual learning on tiny-ImageNet, CIFAR10, and CIFAR100 | test accuracy | Helped |
| Remove regularization (ablation) | Deep Fourier features · Continual learning on tiny-ImageNet, CIFAR10, and CIFAR100 | test accuracy | Hurt |
| Use shallow Fourier features only in the input layer instead of deep Fourier features in every layer (ablation) | Fourier-feature network · Continual learning with random labels | trainability | Hurt |
| Use a single sin activation instead of concatenated sine and cosine activations (ablation) | Fourier-feature network · Continual learning with changing tasks | trainability | Hurt |
| Replace deep Fourier features with CReLU (ablation) | Continual-learning network · Random-label continual learning | trainability | Hurt |
| Replace ReLU with α-linearized ReLU using larger α (ablation) | Continual-learning network · Linearly separable changing-label task | trainability | Helped |
Surprising findings
- Sin activation alone has high unit sign entropy yet still loses trainability, so sign entropy alone is not sufficient.
- Networks with deep Fourier features overfit without regularization but tolerate much stronger regularization, which improves generalization.
Limitations
- Linear and deep linear networks lack expressive power for complex tasks, so their sustained trainability alone is insufficient.
- Deep Fourier features overfit without regularization and rely on spectral regularization for good generalization.
- The linear-approximation guarantee for Fourier features has a bounded error (~0.05) and halves effective width.
Experimental setup
Datasets: CIFAR10 CIFAR100 tiny-ImageNet MNIST Fashion MNIST EMNIST SVHN
Benchmarks: CIFAR10 CIFAR100 tiny-ImageNet MNIST Fashion MNIST EMNIST
Metrics: test accuracy training accuracy (trainability)
Baselines: ReLU L2 regularization towards zero L2 regularization towards initialization spectral regularization Concatenated ReLU Dormant Neuron Recycling Shrink and Perturb Streaming Elastic Weight Consolidation
Builds on: concatenated ReLU spectral regularization Shrink and Perturb L2 regularization towards initialization
Closest papers by meaning
- Spectral Collapse Drives Loss of Plasticity in Deep Continual Learning (ICML 2026 · similarity 0.88)
- Preserving Plasticity in Continual Learning via Dynamical Isometry (ICML 2026 · similarity 0.87)
- Learning Continually by Spectral Regularization (ICLR 2025 · similarity 0.86)
- Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspective (ICML 2025 · similarity 0.85)
- Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learning (ICLR 2024 · similarity 0.85)
- Self-Normalized Resets for Plasticity in Continual Learning (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.