Research map / Continual learning / Online and replay-based
RanDumb: Random Representations Outperform Online Continually Learned Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that a fixed random Fourier feature projection of raw pixels plus an online-updated Mahalanobis nearest-class-mean linear classifier (RanDumb) beats continually learned deep representations in online and low-exemplar continual learning, and that linear classifiers on frozen pretrained features beat most continual fine-tuning and prompt-tuning methods.
Problem
Online continual learning methods jointly train deep representations with classifiers, but their learned representations are rarely checked against simple fixed alternatives, especially in one-pass, low-exemplar and exemplar-free regimes.
Approach
Replace the continually trained deep feature extractor with a fixed random Fourier feature approximation of the RBF kernel on raw pixels, decorrelate features with an online OAS-shrunk covariance (Mahalanobis distance), and classify by nearest class mean, updated one sample at a time with no exemplar storage.
Key results
- On Benchmark A (one class per task), RanDumb reaches 98.3 on MNIST, 55.6 on CIFAR10, 28.6 on CIFAR100 and 17.7 on miniImageNet, beating VAE-GC and most OCL methods, though PEC is better on CIFAR10 (58.9).
- On Benchmark F with ViT-B/16 IN-21K features, RanDumb scores 42.2 on ImageNet-A versus 33.8 for RanPAC-imp, and 92.4 on VTAB, while its Stanford Cars accuracy (67.1) is far above SLCA (18.2).
- RanDumb recovers roughly 70-90% of the performance gap to joint training across online benchmarks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace continually learned representations with fixed random Fourier features and a linear classifier | online continual learning · MNIST, CIFAR-10, CIFAR-100, and miniImageNet (Benchmark A) | accuracy: +5.9; -3.3; +2.1; +2.8 | Mixed |
| replace continually learned representations with fixed random Fourier features and a linear classifier | online continual learning · CIFAR-100 (10/10, Benchmark C) | accuracy: +4.0 | Helped |
| remove decorrelation (ablation) | RanDumb · MNIST (10/1) | accuracy: 83.8 (-14.5) | Helped |
| remove decorrelation (ablation) | RanDumb · CIFAR-10 (10/1) | accuracy: 30.0 (-25.6) | Helped |
| remove random embedding (ablation) | RanDumb · CIFAR-100 (10/1) | accuracy: 19.0 (-9.6) | Helped |
| remove decorrelation and random embedding (ablation) | RanDumb · CIFAR-100 (10/1) | accuracy: 10.4 (-18.2) | Helped |
| add flip augmentation (ablation) | RanDumb · CIFAR-10 | accuracy: +3.1 | Helped |
| add flip augmentation (ablation) | RanDumb · CIFAR-100 | accuracy: +1.7 | Helped |
| add flip augmentation (ablation) | RanDumb · TinyImageNet | accuracy: +0.4 | Helped |
| replace random Fourier features with RP+ReLU (ablation) | RanDumb · CIFAR-10 | accuracy: 55.6 (+6.8) | Helped |
| vary the ridge parameter from 10^{-5} to 10^{-4} (ablation) | RanDumb · CIFAR-10 | accuracy: 55.6 -> 52.6 | Hurt |
| vary the ridge parameter from 10^{-5} to 10^{-4} (ablation) | RanDumb · TinyImageNet | accuracy: 11.1 -> 11.6 | Helped |
Surprising findings
- Random Fourier features on raw pixels outperform continually learned deep representations from state-of-the-art online continual learning methods.
- Prompt-tuning methods collapse under long one-class-per-task sequences, while a frozen-feature linear classifier remains strong.
Limitations
- No theoretical justification for why continual training fails to learn good representations.
- Limited in scope to low-exemplar and online continual learning; not evaluated for high-exemplar or offline settings where GDumb performs better (e.g., CIFAR10 Benchmark D).
- Underperforms PEC on CIFAR10 in Benchmark A and OnPro on CIFAR100 in Benchmark B.2.
Experimental setup
Datasets: MNIST CIFAR10 CIFAR100 TinyImageNet miniImageNet ImageNet-R ImageNet-A CUB Stanford Cars VTAB CIFAR
Benchmarks: MNIST CIFAR10 CIFAR100 miniImageNet TinyImageNet CUB ImageNet-R ImageNet-A VTAB Stanford Cars OB
Metrics: average accuracy after last task
Baselines: VAE-GC PEC GDumb SLCA RanPAC-imp OnPro SEDEM VR-MCL PredKD L2P DualPrompt CODA-Prompt
Builds on: SLDA GDumb RanPAC random Fourier features OAS shrinkage
Closest papers by meaning
- Measuring Representational Shifts in Continual Learning: A Linear Transformation Perspective (ICML 2025 · similarity 0.88)
- Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers (ICML 2025 · similarity 0.88)
- Learning Representations on the Unit Sphere: Investigating Angular Gaussian and Von Mises-Fisher Distributions for Online Continual Learning (AAAI 2024 · similarity 0.87)
- Forgetting, Ignorance or Myopia: Revisiting Key Challenges in Online Continual Learning (NeurIPS 2024 · similarity 0.87)
- Learning to Continually Learn with the Bayesian Principle (ICML 2024 · similarity 0.86)
- Learning Equi-angular Representations for Online Continual Learning (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.