Research map / Continual learning / Online and replay-based
Dealing with Synthetic Data Contamination in Online Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that synthetic (diffusion-generated) image contamination of training data degrades existing online class-incremental learning methods, and proposes ESRM (entropy-based buffer selection plus real-synthetic similarity contrastive loss) that largely mitigates the drop.
Problem
Online continual learning on web-collected datasets is threatened by AI-generated (synthetic) images that contaminate the data stream, degrading the accuracy of existing online CL methods, especially at high contamination ratios.
Approach
ESRM combines Entropy Selection, a replay-buffer management strategy that favors high-entropy (likely real) samples, with Real-synthetic similarity Maximization, a supervised-contrastive-style loss between high-entropy and low-entropy stream groups and between stream and memory, plus self-distillation to reduce overconfidence.
Key results
- On C100/SDXL at 80% contamination, ESRM reaches 44.48% final accuracy versus 33.20% for OnPro and 31.33% for ER (clean CIFAR-100 ER: 38.70%).
- On Tiny/SDXL at 50% contamination, ESRM achieves 29.36% (-2.79 from clean) while baselines drop by roughly 6.9-9.8 points.
- ESRM's drop at 95% contamination on C100/SDXL is -10.27 points versus -14.70 for OnPro and -13.14 for ER.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase synthetic contamination from 0% to 95% | online CL methods · CIFAR-10/SDXL | average accuracy: 63.93{ ±2.40} -> 47.39{ ±2.76} (ER); 74.87{ ±1.58} -> 56.41{ ±2.59} (OnPro) | Hurt |
| Use ESRM instead of the strongest baseline | online CL on C10/SDXL at P=95% · C10/SDXL (P=95%) | average accuracy: 56.41{ ±2.59} -> 60.04{ ±2.29} | Helped |
| Use ESRM instead of the strongest baseline | online CL on C100/SDXL at P=95% · C100/SDXL (P=95%) | average accuracy: 28.84{ ±1.28} -> 37.45{ ±0.56} | Helped |
| Use ESRM instead of the strongest baseline | online CL on Tiny/SDXL at P=95% · Tiny/SDXL (P=95%) | average accuracy: 5.37{ ±0.63} -> 13.02{ ±0.79} | Helped |
| Use ESRM instead of the strongest baseline | online CL on In-100/SDXL at P=95% · In-100/SDXL (P=95%) | average accuracy: 25.14{ ±1.26} -> 32.86{ ±0.99} | Helped |
| Add Entropy Selection to the memory strategy (ablation) | ESRM on C100/SDXL · C100/SDXL (P=70%) | accuracy: 44.84{ ±0.80} -> 45.92{ ±0.42} | Helped |
| Add Entropy Selection to the memory strategy (ablation) | ESRM on C100/SDXL · C100/SDXL (P=80%) | accuracy: 42.62{ ±0.87} -> 44.48{ ±0.41} | Helped |
| Add self-distillation loss (ablation) | ESRM on C100/SDXL · C100/SDXL (P=70%) | accuracy: 44.30{ ±0.78} -> 45.92{ ±0.42} | Helped |
| Add self-distillation loss (ablation) | ESRM on C100/SDXL · C100/SDXL (P=80%) | accuracy: 42.59{ ±0.69} -> 44.48{ ±0.41} | Helped |
| Add RM loss (ablation) | ESRM on C100/SDXL · C100/SDXL (P=70%) | accuracy: 44.03{ ±0.73} -> 45.92{ ±0.42} | Helped |
| Add RM loss (ablation) | ESRM on C100/SDXL · C100/SDXL (P=80%) | accuracy: 42.32{ ±0.52} -> 44.48{ ±0.41} | Helped |
| Use synthetic-only rather than entropy-selected memory (ablation) | ESRM on C100/SDXL · C100/SDXL (P=70%) | accuracy: 45.92{ ±0.42} -> 25.91{ ±0.79} | Hurt |
Surprising findings
- Storing only real samples in the replay buffer (with ground-truth synthetic status) recovers near-clean performance even at 80% contamination, implying the buffer is the key vulnerability.
- Synthetic images are better clustered in embedding space than real images, and entropy is lower for synthetic data, which the authors use to motivate the method.
Limitations
- Only five generative models were used; commercial generators like Midjourney and DALL-E were not tested.
- Synthetic images are generated from simple class-name prompts, not diverse user prompts.
- Entropy-based selection is only moderately discriminative (AUC 0.7098 for real vs synthetic).
Experimental setup
Datasets: CIFAR-10 CIFAR-100 TinyImageNet ImageNet-100 SDXL-generated twin datasets Mix twin datasets (five diffusion models)
Benchmarks: CIFAR-10 CIFAR-100 TinyImageNet ImageNet-100 DIL-CIFAR20
Metrics: final average accuracy learning accuracy relative forgetting
Baselines: ER DER++ ER-ACE OCM GSA OnPro
Builds on: Experience Replay Dark Experience Replay OnPro Reservoir Sampling Supervised contrastive loss
Closest papers by meaning
- On the Stability of Iterative Retraining of Generative Models on their own Data (ICLR 2024 · similarity 0.88)
- The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better (NeurIPS 2024 · similarity 0.86)
- Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferences (NeurIPS 2024 · similarity 0.86)
- Slight Corruption in Pre-training Data Makes Better Diffusion Models (NeurIPS 2024 · similarity 0.86)
- Real-Fake: Effective Training Data Synthesis Through Distribution Matching (ICLR 2024 · similarity 0.86)
- A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspective (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.