Research map / Continual learning / Online and replay-based
ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Think
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ZeroFlow, a benchmark evaluating forward-pass (zeroth-order and forward-gradient) optimizers for overcoming catastrophic forgetting in class-incremental learning under gradient bans, and proposes three ZO enhancements (hybrid ZO, historical gradient mixing, sparsity-induced estimation).
Problem
Catastrophic forgetting must be mitigated when backpropagation/gradients are unavailable (black-box APIs, hardware limits, non-differentiable systems), a setting the authors call "gradient bans".
Approach
Benchmarks seven forward-pass-based optimizers (ZO-SGD, ZO-SGD-Sign, ZO-SGD-Conserve, ZO-Adam variants, and Forward-Grad) against FO SGD/Adam for class-incremental learning on pretrained ViT models, and adds three ZO enhancements that reduce gradient-estimation variance to further reduce forgetting.
Key results
- ZO-SGD and other forward-pass methods are comparable to or surpass FO-SGD/FO-Adam on forgetting metrics; on OmniBenchmark ZO-Adam (Avg 76.15) beats FO-Adam (74.27) for EASE.
- ZO-SGD reduces GPU memory from 12.08 GB (FO-SGD) to 2.41 GB and per-epoch runtime roughly halves on CIFAR-100 (32.4s vs 59.3s).
- Combining hybrid ZO, historical gradient mixing, and sparsity raises ZO-SGD average accuracy on CIFAR-100 from 57.87 to 62.07, above FO-SGD's 61.24.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ZO-SGD instead of FO-SGD | EASE with ViT-B/16 · CIFAR-100 | average accuracy: 91.23 -> 78.62 | Hurt |
| Use ZO-SGD-Sign instead of ZO-SGD | EASE with ViT-B/16 · CIFAR-100 | average accuracy: 78.62 -> 83.21 | Helped |
| Use ZO-SGD-Sign instead of FO-SGD | EASE with ViT-B/16 · CUB | average accuracy: 89.31 -> 89.81 | Helped |
| Use ZO-SGD instead of FO-SGD | EASE with ViT-B/16 · ImageNet-A | average accuracy: 61.24 -> 57.87 | Hurt |
| Use ZO-SGD-Conserve instead of FO-SGD | EASE with ViT-B/16 · OmniBenchmark | average accuracy: 74.73 -> 77.07 | Helped |
| Use ZO-SGD (q=1) instead of FO-SGD | ViT-B/16 training · ImageNet-A | GPU memory: 12.08 GB -> 2.41 GB | Helped |
| Use ZO-SGD (q=1) instead of FO-SGD | ViT-B/16 training · ImageNet-A | runtime per epoch: 12.2s -> 6.8s | Helped |
| Add 20% historical estimation (ablation) | ZO-SGD · ImageNet-A | average accuracy: 57.87 -> 58.90 | Helped |
| Add historical estimation at 40%, 60%, or 80% (ablation) | ZO-SGD · ImageNet-A | average accuracy: 57.87 -> 58.76; 57.87 -> 58.34; 57.87 -> 57.83 | Helped |
| Add 90% sparsity-induced estimation (ablation) | ZO-SGD · ImageNet-A | average accuracy: 57.87 -> 59.47 | Helped |
| Add sparsity-induced estimation at 10% through 80% (ablation) | ZO-SGD · ImageNet-A | average accuracy: 57.87 -> 59.17; 57.87 -> 59.46; 57.87 -> 59.29; 57.87 -> 59.39; 57.87 -> 59.45; 57.87 -> 59.26; 57.87 -> 59.39; 57.87 -> | Helped |
| Add sparsity-induced estimation at 10% through 90% (ablation) | ZO-SGD · ImageNet-A | last-task accuracy: 48.32 -> 48.58; 48.32 -> 49.05; 48.32 -> 48.72; 48.32 -> 48.91; 48.32 -> 49.24; 48.32 -> 49.11; 48.32 -> 49.05; 48.32 -> | Helped |
Surprising findings
- Forward-pass-only ZO optimization matches or outperforms backpropagation-based FO training for forgetting mitigation in the benchmarked settings.
- Plain ZO-SGD with a single query lowers memory and runtime while the ZO-SGD-Sign variant with the same query budget is comparably efficient.
Limitations
- ZO optimization converges slowly with parameter dimension, so the study focuses on pretrained-backbone continual learning rather than training from scratch.
- High variance of ZO gradient estimates with a single query; more queries improve stability but raise runtime substantially.
- Evaluation limited to class-incremental learning on vision transformers with ViT-B/16 and a few datasets; no LLM experiments.
Experimental setup
Datasets: ImageNet-1K CIFAR-100 CUB ImageNet-A OmniBenchmark
Benchmarks: CIFAR-100 CUB ImageNet-A OmniBenchmark
Metrics: average accuracy last accuracy forgetting FWT BWT GPU memory runtime
Baselines: FO-SGD FO-Adam ZO-SGD ZO-SGD-Sign ZO-SGD-Conserve Forward-Grad MEMO
Builds on: SPSA ZO-SGD ZO-AdaMM EASE APER MeZO
Closest papers by meaning
- Stabilizing Zero-Shot Prediction: A Novel Antidote to Forgetting in Continual Vision-Language Tasks (NeurIPS 2024 · similarity 0.90)
- On the Theory of Continual Learning with Gradient Descent for Neural Networks (ICML 2026 · similarity 0.89)
- STAR: Stability-Inducing Weight Perturbation for Continual Learning (ICLR 2025 · similarity 0.88)
- Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learning (ICLR 2024 · similarity 0.87)
- Predicting the Susceptibility of Examples to Catastrophic Forgetting (ICML 2025 · similarity 0.87)
- Mixture of Experts Meets Prompt-Based Continual Learning (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.