Research map / Continual learning / Online and replay-based
Learning Representations on the Unit Sphere: Investigating Angular Gaussian and Von Mises-Fisher Distributions for Online Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MAP-based representation learning with angular Gaussian and von Mises-Fisher losses using fixed one-hot class directions and a memory replay for online class-incremental continual learning.
Problem
Online class-incremental continual learning, where data arrive as a single-pass stream without task boundaries in realistic (blurry) settings, and contrastive methods need large batches and distillation methods need task boundaries.
Approach
Derives losses from MAP estimation with representations on the unit sphere, using angular Gaussian (projected Gaussian) and von Mises-Fisher class-conditional densities, with class means fixed to standard basis vectors so training does not depend on batch statistics or data drift.
Key results
- AGD-FD reaches 36.32% on CIFAR100 (M=1k) with blurry boundaries vs 30.81% for SCR (best baseline there), and 50.84% at M=5k.
- Under clear boundaries on CIFAR100 with M=5k, AGD-FD achieves 50.54% vs OCM 42.49%, an improvement of about 8 points.
- On CIFAR10 with clear boundaries, OCM (72.6%) slightly outperforms AGD-FD (70.06%) at M=1k.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AGD-FD instead of the strongest reported baseline | Online continual learning · CIFAR100 clear, M=5k | final average accuracy (%): 42.49{ ±1.45} -> 50.54{ ±0.67} | Helped |
| Use AGD-FD instead of the strongest reported baseline | Online continual learning · Tiny ImageNet clear, M=10k | final average accuracy (%): 32.3{ ±1.34} -> 34.77{ ±0.52} | Helped |
| Use AGD-FD instead of the strongest reported baseline | Online continual learning · CIFAR10 clear, M=1k | final average accuracy (%): 72.6{ ±1.98} -> 70.06{ ±1.11} | Hurt |
| Use AGD-FD instead of the strongest reported baseline | Online continual learning · CIFAR10 blurry, M=1k | final average accuracy (%): 68.31{ ±1.61} -> 71.59{ ±0.99} | Helped |
| Use AGD-FD instead of the strongest reported baseline | Online continual learning · CIFAR100 blurry, M=5k | final average accuracy (%): 41.16{ ±1.57} -> 50.84{ ±0.72} | Helped |
| Use AGD-FD instead of the strongest reported baseline | Online continual learning · Tiny ImageNet blurry, M=10k | final average accuracy (%): 31.6{ ±1.01} -> 35.96{ ±0.47} | Helped |
| Use AGD-FD instead of SCR | Online continual learning · CORe50 new instances, M=5k | final average accuracy: 48.6±2.3 -> 50.0±0.9 | Helped |
| Use fixed class directions e_c instead of estimating spherical means (ablation) | AGD-FD · CIFAR100, M=5k, 10 tasks, retrieval batch 64 | final AA: 25.54±1.48 -> 50.36±0.58 | Helped |
| Use fixed class directions e_c instead of estimating spherical means (ablation) | AGD-FD · CIFAR100, M=5k, 10 tasks, retrieval batch 512 | final AA: 45.79±0.79 -> 49.62±0.49 | Helped |
| Increase multi-view batch training from 3 to 10 views (ablation) | AGD-FD · CIFAR100, M=5k, 10 tasks | accuracy: 49.4±0.4 -> 51.2±0.4 | Helped |
| Apply multi-view training to SCR (ablation) | SCR · CIFAR100, M=5k, 10 tasks | accuracy: 43.9±0.6 -> 46.5±0.6 | Helped |
| Use AGD-FD rather than vMF-FD | Fixed-direction spherical representation learning · CIFAR100 clear, M=5k | final average accuracy (%): 50.39{ ±0.75} -> 50.54{ ±0.67} | No clear effect |
Surprising findings
- Fixing class means to one-hot basis vectors greatly outperforms estimating spherical batch means, even with larger memory retrieval batches (e.g., 50.36% vs 25.54% at |X_M|=64).
Limitations
- On CIFAR10 with clear boundaries, OCM outperforms the proposed method.
- Assumes i.i.d. components with diagonal covariance; non-diagonal covariance not explored.
- Distillation-style baselines were adapted to blurry boundaries with heuristic task-change detection.
Experimental setup
Datasets: CIFAR10 CIFAR100 Tiny ImageNet CORe50
Benchmarks: CIFAR10 CIFAR100 Tiny ImageNet CORe50
Metrics: final average accuracy
Baselines: ER SCR GDumb AGEM DER++ DVC ER-ACE OCM PFC
Builds on: Experience Replay SCR Guillotine Regularization PFC
Closest papers by meaning
- RanDumb: Random Representations Outperform Online Continually Learned Representations (NeurIPS 2024 · similarity 0.87)
- Forgetting, Ignorance or Myopia: Revisiting Key Challenges in Online Continual Learning (NeurIPS 2024 · similarity 0.86)
- Measuring Representational Shifts in Continual Learning: A Linear Transformation Perspective (ICML 2025 · similarity 0.86)
- Learning to Continually Learn with the Bayesian Principle (ICML 2024 · similarity 0.86)
- Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers (ICML 2025 · similarity 0.86)
- Task-Free Continual Generation and Representation Learning via Dynamic Expansionable Memory Cluster (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.