Research map / Continual learning / Online and replay-based
Make Continual Learning Stronger via C-Flat
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes C-Flat, a zeroth-plus-first-order sharpness-aware optimizer plug-and-play into class-incremental learning methods, improving average accuracy across seven CIL baselines on CIFAR-100, ImageNet-100 and Tiny-ImageNet.
Problem
Continual learning suffers from catastrophic forgetting due to the sensitivity-stability dilemma; zeroth-order sharpness-aware minimization may converge to sharp minima rather than flat, generalizable ones in the joint old+new knowledge space.
Approach
Combine zeroth-order sharpness (SAM-style max loss in a neighborhood) with a first-order gradient-norm flatness regularizer, implemented via Hessian-vector products, as a drop-in optimizer for any CL method.
Key results
- C-Flat improves average accuracy over all 7 CIL baselines in most settings, with average return of +0.62% to +1.34% across the six incremental scenarios.
- Maximum Hessian eigenvalue for MEMO on CIFAR-100 drops from 67.48 (SGD) to 28.11 at epoch 50 with C-Flat.
- Replacing FS zeroth-order flatness with C-Flat in FS-DGPM series improves accuracy by +0.35 to +0.79 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add C-Flat to WA | WA · CIFAR-100 B0_Inc5 | average accuracy (%): 63.36 -> 65.70 | Helped |
| Add C-Flat to DER | DER · CIFAR-100 B0_Inc10 | average accuracy (%): 71.01 -> 72.08 | Helped |
| Add C-Flat to Replay | Replay · ImageNet-100 B50_Inc10 | average accuracy (%): 63.89 -> 63.60 | Hurt |
| Add C-Flat to iCaRL | iCaRL · ImageNet-100 B50_Inc25 | average accuracy (%): 77.25 -> 76.22 | Hurt |
| Add C-Flat to FOSTER | FOSTER · ImageNet-100 B50_Inc25 | average accuracy (%): 87.81 -> 87.81 | No clear effect |
| Replace FS with C-Flat | FS-DGPM · GPM benchmark (not specified in excerpt) | accuracy: 73.14 -> 73.72; +0.58 | Helped |
| Apply C-Flat in only 10% of iterations (ablation) | Replay · CIFAR-100 B0_Inc20 | accuracy: 60.28 -> 60.48 | Helped |
| Apply C-Flat in only 10% of iterations (ablation) | iCaRL · CIFAR-100 B0_Inc20 | accuracy: 62.74 -> 62.75 | Helped |
| Apply C-Flat in only 10% of iterations (ablation) | WA · CIFAR-100 B0_Inc20 | accuracy: 67.75 -> 68.02 | Helped |
| Apply C-Flat in only 10% of iterations (ablation) | PODNet · CIFAR-100 B0_Inc20 | accuracy: 63.05 -> 63.80 | Helped |
| Apply C-Flat in only 10% of iterations (ablation) | DER · CIFAR-100 B0_Inc20 | accuracy: 71.52 -> 71.44 | No clear effect |
| Apply C-Flat in only 10% of iterations (ablation) | FOSTER · CIFAR-100 B0_Inc20 | accuracy: 69.30 -> 69.71 | Helped |
Surprising findings
- C-Flat applied at only 10% of iterations still improves CL performance, according to the appendix ablation.
- ImageNet-100 B50_Inc25 with FOSTER shows no gain from C-Flat (87.81 both).
Limitations
- Zeroth-order sharpness and first-order flatness terms double the per-iteration cost relative to SGD.
- No error bars or statistical significance reported across runs.
- Evaluation limited to class-incremental learning on image classification benchmarks.
Experimental setup
Datasets: CIFAR-100 ImageNet-100 Tiny-ImageNet
Benchmarks: CIFAR-100 ImageNet-100 Tiny-ImageNet
Metrics: average accuracy across phases Hessian eigenvalues Hessian trace loss landscape visualization
Baselines: Replay iCaRL WA PODNet DER FOSTER MEMO SAM GAM SGD
Builds on: sharpness-aware minimization GAM iCaRL WA DER FOSTER MEMO PODNet GPM DGPM
Closest papers by meaning
- Zeroth-Order Sharpness-Aware Learning with Exponential Tilting (NeurIPS 2026 · similarity 0.86)
- Model Sensitivity Aware Continual Learning (NeurIPS 2024 · similarity 0.86)
- Zeroth-Order Optimization Finds Flat Minima (NeurIPS 2025 · similarity 0.85)
- Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late In Training (ICLR 2025 · similarity 0.85)
- Stable Minima of ReLU Neural Networks Suffer from the Curse of Dimensionality: The Neural Shattering Phenomenon (NeurIPS 2025 · similarity 0.85)
- Sharpness-Aware Minimization Enhances Feature Quality via Balanced Learning (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.