Research map / Continual learning / Online and replay-based
On Local Overfitting and Forgetting in Deep Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a forget-fraction score to detect "local overfitting" (forgetting of validation points late in training) and a checkpoint-fusion method (KnowledgeFusion) that recovers forgotten knowledge from a single training run, optionally distilled into one model.
Problem
Deep networks rarely show classical overfitting as validation accuracy drops, yet they still forget subsets of validation data during training; the paper asks whether this local overfitting can be detected and exploited to improve final accuracy.
Approach
Define a forget fraction F_e (test points correct at epoch e but wrong at the end) to capture local overfitting even when test accuracy rises monotonically; analyze forgetting in deep linear networks via GD dynamics and PCA; then fuse the final checkpoint with a validation-selected mid-training checkpoint (with a window) and iterate, using only one training run.
Key results
- KnowledgeFusion (infinite budget) improves ResNet18 on CIFAR-100 from 78.07% to 79.13% and on TinyImagenet from 64.95% to 68.5%.
- On ImageNet, KF improves ConvNeXt large from 82.92% to 83.96% and ViT16 base from 79.16% to 80.34%.
- Under 20%/40% symmetric label noise on CIFAR-100, KF gains 7.4 and 10.1 points over the single network.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply KF to the trained ResNet18 | ResNet18 · CIFAR-100 | test accuracy: 78.07 ± .28 -> 79.13 ± .14 | Helped |
| Apply KF to the trained ResNet18 | ResNet18 · TinyImageNet | test accuracy: 64.95 ± .24 -> 68.5 ± .36 | Helped |
| Apply KF to the trained network | ImageNet classifiers · ImageNet | test accuracy: 75.74 ± .14 -> 76.52 ± .16; 82.92 ± .11 -> 83.96 ± .09; 79.16 ± .1 -> 80.34 ± .08; 82.51 ± .15 -> 83.81 ± .14 | Helped |
| Apply KF to the trained ResNet18 | ResNet18 · CIFAR-100 asymmetric, 40% label noise | test accuracy: 49.4 ± .3 -> 62.1 ± .5 | Helped |
| Apply KF to the trained model (ablation) | ImageNet-pretrained ResNet18 fine-tuned on CIFAR-100 · CIFAR-100 | test accuracy: 80.72 ± .53 -> 81.64 ± .27 | Helped |
| Apply KF to the trained model (ablation) | ImageNet-pretrained ResNet18 with only the head fine-tuned · TinyImageNet | test accuracy: 54.78 ± .13 -> 59.5 ± .03 | Helped |
| Use KF instead of EMA | ResNet18 · TinyImageNet | improvement in test accuracy: 0.73 ± .11 -> 3.54 ± .14; 2.51 ± .01 -> 3.54 ± .14 | Helped |
| Add test-time augmentation to KF (ablation) | ResNet18 · CIFAR-100, 0% label noise | test accuracy: 78.71 ± .2 -> 79.55 ± .3 | Helped |
| Increase model size from small to large while using KF (ablation) | ConvNeXt on ImageNet · ImageNet | test accuracy: 83.17 ± .04 -> 83.96 ± .09 | Helped |
| Use KF rather than a single network (ablation) | ConvNeXt on ImageNet · ImageNet | test accuracy: 83.21 ± .01 -> 83.17 ± .04 | No clear effect |
| Use a window greater than zero around selected checkpoints (ablation) | Knowledge Fusion · Evaluated scenarios | test accuracy | Helped |
| Use only 5-10% of past checkpoints (ablation) | Knowledge Fusion · Evaluated datasets | test accuracy improvement over a single network: 5-10% | No clear effect |
Surprising findings
- Forgetting of validation points occurs even when test accuracy increases monotonically, i.e., without classical overfitting.
- In epoch-wise double descent, most newly memorized points in the second rise were previously misclassified rather than the forgotten ones being re-learned.
Limitations
- Requires a labeled validation set and multiple saved checkpoints.
- Theoretical analysis is limited to deep linear models and correlation with real networks is empirical.
- Gains on standard clean datasets are modest (around 1%).
Experimental setup
Datasets: CIFAR-100 TinyImagenet ImageNet Animal10N
Benchmarks: CIFAR-100 TinyImagenet ImageNet Animal10N
Metrics: test accuracy amplification bias
Baselines: single network horizontal ensemble fixed jumps snapshot ensemble SWA FGE EMA early stopping test time augmentation
Builds on: self-distillation snapshot ensemble stochastic weight averaging fast geometric ensembling epoch-wise double descent
Closest papers by meaning
- On the Over-Memorization During Natural, Robust and Catastrophic Overfitting (ICLR 2024 · similarity 0.88)
- The Effectiveness of Random Forgetting for Robust Generalization (ICLR 2024 · similarity 0.87)
- Forgetting Has Neighbors: Localized Collateral Forgetting in Machine Unlearning (NeurIPS 2026 · similarity 0.86)
- How Much Can We Forget about Data Contamination? (ICML 2025 · similarity 0.86)
- Rethinking Benign Overfitting in Two-Layer Neural Networks (ICML 2025 · similarity 0.86)
- Label-Agnostic Forgetting: A Supervision-Free Unlearning in Deep Models (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.