Research map / Continual learning / Online and replay-based
Summarizing Stream Data for Memory-Constrained Online Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Summarizing Stream Data (SSD), which distills stream mini-batches into a small set of synthetic memory images via gradient matching plus past-assisted relationship matching, improving replay for memory-constrained online class-incremental continual learning.
Problem
Online class-incremental continual learning with replay suffers when the memory is very small, because each stored original image carries limited information.
Approach
Instead of selecting original stream images for the replay memory, the memory is filled with synthetic summarized images that are optimized by dataset-distillation-style gradient matching against the stream, with a past-assisted summarizing model (using real memory images) and a relationship-matching loss anchored on past summarized samples.
Key results
- On sequential CIFAR-100 with memory size 100, SSD on SCR reaches 12.1% average end accuracy versus 9.0% for SCR (+3.1).
- SSD applied to ER improves CIFAR-100 accuracy at K=100/500/1000 by +1.2/+1.5/+2.8 points.
- Ablation shows dynamic memory, summarizing with gradient matching, and past assistance add cumulatively, reaching +3.1 over SCR at K=100.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace selected original memory images with summarized stream-data samples using SSD | SCR · CIFAR-100, K=100/500/1000 | average end accuracy: 9.0 -> 12.1; 20.6 -> 23.0; 26.6 -> 28.8 | Helped |
| add dynamic memory to SCR (ablation) | SCR · CIFAR-100, K=100/500/1000 | average end accuracy: 9.0 -> 10.7; 20.6 -> 21.3; 26.6 -> 27.2 | Helped |
| add stream-data summarizing with gradient matching to dynamic memory (ablation) | SCR with dynamic memory · CIFAR-100, K=100/500/1000 | average end accuracy: 10.7 -> 11.5; 21.3 -> 22.3; 27.2 -> 28.2 | Helped |
| add past assistance to summarizing (ablation) | SCR with dynamic memory and summarizing · CIFAR-100, K=100/500/1000 | average end accuracy: 11.5 -> 12.1; 22.3 -> 23.0; 28.2 -> 28.8 | Helped |
| apply SSD memory summarization | ER · CIFAR-100, K=100/500/1000 | average end accuracy: 6.3 -> 7.5; 9.3 -> 10.8; 10.1 -> 12.9 | Helped |
| apply SSD memory summarization | DVC · CIFAR-100, K=100/500/1000 | average end accuracy: 10.4 -> 12.0; 16.6 -> 18.5; 20.2 -> 21.8 | Helped |
| replace SCR memory construction with GSS | SCR · CIFAR-100, K=100/500/1000 | average end accuracy: +0.9; -1.3; -2.2 | Mixed |
| increase replays per iteration to 5 (ablation) | SSD · CIFAR-100, K=100/500/1000 | average end accuracy: 12.1 -> 13.5; 23.0 -> 24.4; 28.8 -> 30.4 | Helped |
| increase replays per iteration to 10 (ablation) | SSD · CIFAR-100, K=100/500/1000 | average end accuracy: 12.1 -> 13.7; 23.0 -> 24.7; 28.8 -> 31.1 | Helped |
| distill using more classes together rather than many narrow class slices (ablation) | dataset distillation · CIFAR-100, 50 images per class | accuracy: 51.6; 51.2; 50.7; 48.9 | Mixed |
| set relationship-matching coefficient to γ=1 (ablation) | SSD · online continual learning benchmarks | average end accuracy | Helped |
| increase summarizing interval from 2 to 6 (ablation) | SSD · online continual learning benchmarks | average end accuracy | No clear effect |
Surprising findings
- Summarized images provide gains over SCR and ER, but SSD's gains over GSS-style selection are negative at larger memory sizes for GSS, indicating selection-based memory construction can hurt at larger K.
Limitations
- Summarization optimizes raw image pixels directly, so optimization difficulty grows with memory size and image resolution.
- Summarizing is done only every tau iterations, and performance drops relative to every-iteration summarizing (trade-off with compute).
Experimental setup
Datasets: CIFAR-100 Mini-ImageNet Tiny-ImageNet
Benchmarks: Sequential CIFAR-100 Sequential Mini-ImageNet Sequential Tiny-ImageNet
Metrics: average end accuracy average forgetting
Baselines: fine-tune iid offline EWC++ LwF AGEM ER MIR GSS ASER DVC SCR OCM
Builds on: SCR dataset condensation with gradient matching (DC) ER RAR
Closest papers by meaning
- Beyond Buffer Limits: Energy-Based Data Reassembly for Continual Learning (ICML 2026 · similarity 0.91)
- Budgeted Online Continual Learning by Adaptive Layer Freezing and Frequency-based Sampling (ICLR 2025 · similarity 0.89)
- Online Curvature-Aware Replay: Leveraging $\mathbf{2^{nd}}$ Order Information for Online Continual Learning (ICML 2025 · similarity 0.89)
- Layerwise Proximal Replay: A Proximal Point Method for Online Continual Learning (ICML 2024 · similarity 0.88)
- Online Task-Free Continual Learning via Dynamic Expansionable Memory Distribution (CVPR 2025 · similarity 0.87)
- Do Your Best and Get Enough Rest for Continual Learning (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.