Research map / Continual learning / Online and replay-based
Learning Equi-angular Representations for Online Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EARL, an online continual learning method that induces neural collapse to a fixed simplex ETF classifier using preparatory (rotation-transformed) data training and inference-time residual correction to improve anytime accuracy under single-pass training.
Problem
Online continual learning with single-pass training underfits because the model does not reach the terminal phase of training, so neural collapse to an ETF structure does not form quickly enough, hurting anytime inference accuracy.
Approach
Use a fixed ETF classifier with K=d+1 vectors, train on rotated (negative-transformation) replay samples assigned to unseen classes to prevent new-class features from being biased toward old classes, and at inference add residuals stored during training from a feature-residual memory (kNN-weighted) to correct the output.
Key results
- On CIFAR-10 disjoint, EARL reaches 78.31 AAUC vs 77.59 for the best baseline X-DER.
- On ImageNet-1K disjoint, EARL achieves 34.33 AAUC, about +4.0 over the best prior baseline (REMIND 32.47 excluding MEMO-style gains; reported as +4.0% over the next best).
- Ablation on CIFAR-10 disjoint shows preparatory data training gives the larger gain: removing both components drops AAUC from 78.61 to 74.77.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add residual correction (ablation) | EARL · CIFAR-10 Disjoint | AAUC: 77.78±0.52 -> 78.61±0.72 | Helped |
| Add residual correction (ablation) | EARL · CIFAR-10 Gaussian-Scheduled | AAUC: 68.37±0.10 -> 69.62±0.19 | Helped |
| Add residual correction (ablation) | EARL · CIFAR-100 Disjoint | AAUC: 56.28±1.40 -> 57.42±1.24 | Helped |
| Add residual correction (ablation) | EARL · CIFAR-100 Gaussian-Scheduled | AAUC: 47.05±0.71 -> 48.19±0.61 | Helped |
| Add preparatory data training (PDT) and residual correction (RC) (ablation) | EARL · CIFAR-10 Disjoint | AAUC: 74.77±0.77 -> 78.61±0.72 | Helped |
| Add preparatory data training (PDT) and residual correction (RC) (ablation) | EARL · CIFAR-10 Gaussian-Scheduled | AAUC: 65.60±0.25 -> 69.62±0.19 | Helped |
| Add preparatory data training (PDT) and residual correction (RC) (ablation) | EARL · CIFAR-100 Disjoint | AAUC: 52.91±1.05 -> 57.42±1.24 | Helped |
| Add preparatory data training (PDT) and residual correction (RC) (ablation) | EARL · CIFAR-100 Gaussian-Scheduled | AAUC: 44.34±0.55 -> 48.19±0.61 | Helped |
| Use EARL instead of the strongest listed baseline | Online continual learning methods · CIFAR-10 Disjoint | AAUC: 77.59±0.62 -> 78.31±0.72 | Helped |
| Use EARL instead of the strongest listed baseline | Online continual learning methods · CIFAR-10 Gaussian-Scheduled | AAUC: 65.46±0.46 -> 69.52±0.19 | Helped |
| Use EARL instead of the strongest listed baseline | Online continual learning methods · CIFAR-100 Disjoint | AAUC: 52.95±1.25 -> 57.12±1.22 | Helped |
| Use EARL instead of the strongest listed baseline | Online continual learning methods · CIFAR-100 Gaussian-Scheduled | AAUC: 41.94±0.57 -> 47.89±0.61 | Helped |
Surprising findings
- Preparatory data training contributes more than residual correction in the ablation, with residual correction giving smaller gains.
- EARL does not beat MEMO on ImageNet-1K disjoint Alast, where it is reported as not the best.
Limitations
- The ETF classifier size is bounded by the embedding dimension (K = d+1), limiting the number of classes unless the ETF is expanded.
- Results on ImageNet-1K are from a single seed due to computational constraints.
- Not the best on ImageNet-1K disjoint Alast, where MEMO leads in the reported table.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 TinyImageNet ImageNet-200 ImageNet-1K
Benchmarks: CIFAR-10 CIFAR-100 TinyImageNet ImageNet-200 ImageNet-1K
Metrics: AAUC Alast
Baselines: EWC ER ER-MIR REMIND DER++ SCR ODDL MEMO X-DER
Builds on: neural collapse dot regression loss (Yang et al.) ResMem Greedy Balanced Sampling NC-FSCIL
Closest papers by meaning
- RanDumb: Random Representations Outperform Online Continually Learned Representations (NeurIPS 2024 · similarity 0.86)
- Towards Continual Learning Desiderata via HSIC-Bottleneck Orthogonalization and Equiangular Embedding (AAAI 2024 · similarity 0.86)
- Learning Representations on the Unit Sphere: Investigating Angular Gaussian and Von Mises-Fisher Distributions for Online Continual Learning (AAAI 2024 · similarity 0.85)
- Layerwise Proximal Replay: A Proximal Point Method for Online Continual Learning (ICML 2024 · similarity 0.85)
- Budgeted Online Continual Learning by Adaptive Layer Freezing and Frequency-based Sampling (ICLR 2025 · similarity 0.85)
- Non-exemplar Online Class-Incremental Continual Learning via Dual-Prototype Self-Augment and Refinement (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.