Research map / Continual learning / Online and replay-based
Compact Memory for Continual Logistic Regression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds compact memory for continual logistic regression by estimating K-prior memory vectors and weights via Hessian matching solved with probabilistic PCA (EM), outperforming experience replay at small memory sizes.
Problem
Continual learning with shallow models (logistic regression) suffers catastrophic forgetting, and existing compact memory methods (K-prior optimal memory) require access to all past data to construct, so a practical way to build compact memory online is needed.
Approach
Reformulate the optimal K-prior memory search as Hessian matching at the new task's solution and estimate memory vectors and weights online with a PPCA maximum-likelihood EM update (extended to logistic regression via sigmoid/softmax derivative weights), so no past data is needed.
Key results
- On Split-ImageNet-1000, the method reaches about 60% average accuracy with memory equal to 0.3% of data, versus about 30% for experience replay.
- With 2% memory on Split-ImageNet-1000, accuracy reaches 74%, close to the batch accuracy of 77.6%.
- On USPS, the method with 1% memory outperforms K-prior and replay and approaches batch training; on Split-MNIST with 160 memory it gets 0.820 vs SVD 0.815 vs batch 0.849.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Hessian-matched compact memory instead of replay | Continual logistic regression · Split-ImageNet-1000 (0.3% memory) | accuracy: 30% -> 60% | Helped |
| increase compact memory from 0.3% to 2% of data | Continual logistic regression · Split-ImageNet-1000 | accuracy: 60% -> 74% | Helped |
| use compact memory instead of SVD | Continual multi-output regression · Split-MNIST, memory sizes 10, 20, and 40 | accuracy: 0.639 ± 0.002 -> 0.697 ± 0.004; 0.685 ± 0.003 -> 0.744 ± 0.001; 0.745 ± 0.001 -> 0.798 ± 0.000 | Helped |
| use compact memory instead of SVD | Continual multi-output regression · Split-MNIST, memory size 320 | accuracy: 0.828 ± 0.002 -> 0.815 ± 0.001 | Hurt |
| set PPCA noise ε to 10⁻³ instead of 10⁻⁴ (ablation) | Compact memory · USPS odd vs even, 1% memory | average accuracy: 0.840 ± 0.007 -> 0.820 ± 0.013 | Hurt |
| set PPCA noise ε to 10⁻⁵ instead of 10⁻⁴ (ablation) | Compact memory · USPS odd vs even, 1% memory | average accuracy: 0.840 ± 0.007 -> 0.841 ± 0.008 | No clear effect |
| vary PPCA noise ε among 10⁻³, 10⁻⁴, and 10⁻⁵ (ablation) | Compact memory · USPS odd vs even, 2% and 5% memory | average accuracy: 0.875 ± 0.003; 0.881 ± 0.004; 0.883 ± 0.003; 0.915 ± 0.002; 0.918 ± 0.002; 0.913 ± 0.002 | Mixed |
Surprising findings
- On Split-MNIST, the EM method beats the SVD oracle at small memory sizes (10, 20, 40) but SVD is slightly better at 320 memory.
Limitations
- Limited to last-layer logistic regression with a frozen feature extractor; not yet applied to deep networks.
- Noise hyperparameter epsilon is fixed and chosen by grid search rather than estimated from PPCA.
- Gradient reconstruction is approximate, not perfect, so accuracy gap to batch remains at small memory sizes.
Experimental setup
Datasets: Four-moon USPS Split-MNIST Permuted-MNIST CIFAR-10 CIFAR-100 TinyImageNet-200 ImageNet-1000
Benchmarks: Split-ImageNet-1000 Split-CIFAR-10 Split-CIFAR-100 Split-TinyImageNet-200 USPS odd vs even Split-MNIST Four-moon
Metrics: average accuracy over tasks test accuracy running time ratio
Baselines: K-prior experience replay SVD (optimal memory) batch training
Builds on: K-prior probabilistic PCA online SVD experience replay
Closest papers by meaning
- Do Your Best and Get Enough Rest for Continual Learning (CVPR 2025 · similarity 0.88)
- An Effective Dynamic Gradient Calibration Method for Continual Learning (ICML 2024 · similarity 0.87)
- Learning to Continually Learn with the Bayesian Principle (ICML 2024 · similarity 0.87)
- Summarizing Stream Data for Memory-Constrained Online Continual Learning (AAAI 2024 · similarity 0.87)
- Budgeted Online Continual Learning by Adaptive Layer Freezing and Frequency-based Sampling (ICLR 2025 · similarity 0.87)
- Task-Free Continual Generation and Representation Learning via Dynamic Expansionable Memory Cluster (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.