Research map / Architectures and efficiency / RNNs and normalization
Instance-dependent Early Stopping
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Instance-dependent Early Stopping (IES), which removes "mastered" training instances from backpropagation using a unified threshold on the second-order difference of per-instance loss, cutting backprop instances by 10-50% with comparable accuracy.
Problem
Conventional early stopping halts training for all instances at once, so well-learned instances keep consuming backpropagation compute; per-instance loss thresholds are hard to set uniformly because optimal losses differ across instances.
Approach
Mark an instance as mastered when the absolute second-order difference of its loss across three consecutive epochs falls below a small threshold delta, and exclude mastered instances from backpropagation (the mastered check needs only forward passes). The second-order difference has a lower coefficient of variation than raw loss, so one unified threshold works across instances.
Key results
- IES-2nd saves on average 40.7% of mini-batches on CIFAR-10 (~1.4x wall-time speedup) with test accuracy within about 0.3% of no-removal (e.g., ResNet-18 92.9% vs 92.9%).
- On ImageNet-1k pretraining, IES-pretrained ResNet-101 fine-tuned for 1 epoch gains 0.9% on CIFAR-10, 1.9% on CIFAR-100 and 2.5% on Caltech-101 over the no-removal baseline (per the paper's text).
- On PASCAL VOC, IES saves 20.0% of mini-batches for Faster R-CNN with unchanged mAP (70.2%) and 14.0% for DeepLab v3 with mIoU 76.1% vs 76.2%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| exclude mastered instances using second-order loss differences | ResNet models on CIFAR-10 · CIFAR-10 | test accuracy: 92.9%±0.1% -> 92.9%±0.2%; 93.3%±0.1% -> 93.1%±0.1%; 90.9%±0.2% -> 90.7%±0.2% | No clear effect |
| exclude mastered instances using second-order loss differences | CIFAR-100 models · CIFAR-100 | test accuracy: 69.8%±0.4% -> 69.6%±0.3%; 71.9%±0.5% -> 72.2%±0.5%; 73.4%±0.0% -> 73.3%±0.2% | Mixed |
| exclude mastered instances using second-order loss differences | SGD training on CIFAR-10 and CIFAR-100 · CIFAR-10 and CIFAR-100 | test accuracy: 92.1%±0.1% -> 92.4%±0.0%; 71.4%±0.5% -> 72.3%±0.4% | Mixed |
| pretrain with IES rather than No Removal | ResNet-101 · ImageNet-1k to CIFAR-10, 1-epoch fine-tuning | accuracy: 80.3%$\pm$0.2% -> 81.2%$\pm$0.1% | Helped |
| use IES rather than conventional early stopping | CIFAR-10 training at approximately 2.0× speedup · CIFAR-10 | accuracy: 90.4%$\pm$0.5% -> 93.7%$\pm$0.4% | Helped |
| use IES rather than the strongest listed baseline, EfficientTrain | training at approximately 2.0× speedup · CIFAR-10 | accuracy: 91.5%$\pm$0.2% -> 93.7%$\pm$0.4% | Helped |
| use IES rather than the strongest listed baseline, EfficientTrain | training at approximately 2.0× speedup · CIFAR-100 | accuracy: 75.0%$\pm$0.1% -> 74.9%$\pm$0.5% | No clear effect |
| use the second-order difference criterion instead of other IES criteria (ablation) | IES · CIFAR-10 and CIFAR-100 | test accuracy | Helped |
| vary the IES threshold δ across four orders of magnitude (ablation) | IES · CIFAR-10 and CIFAR-100 | test accuracy: approximately 2% | No clear effect |
| randomly remove training samples | sample-selection training · CIFAR-10 and CIFAR-100 | test accuracy | Hurt |
| remove small-loss samples and rescale remaining gradients | sample-selection training · CIFAR-10 and CIFAR-100 | test accuracy | Mixed |
| exclude mastered instances using IES | Faster R-CNN · PASCAL VOC2007 object detection | mAP: 70.2 $\pm$ 0.2 -> 70.2 $\pm$ 0.1 | No clear effect |
Surprising findings
- Under noisy-label settings with standard early stopping, IES saves 0% of mini-batches and degenerates to the baseline, since no instance becomes mastered before memorization.
- Excluding mastered instances from backpropagation leaves the number of mastered instances nearly unchanged, suggesting models retain learned status without repeated training on them.
Limitations
- No comprehensive theoretical analysis of why the second-order difference is the best removal criterion.
- Mastered criterion fails to trigger under label noise, so IES reduces to the baseline in noisy-label settings.
- Fairness impact is only partially evaluated in the appendix and not thoroughly investigated.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-1k Caltech-101 PASCAL VOC
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-1k CIFAR-10 transfer CIFAR-100 transfer Caltech-101 transfer PASCAL VOC2007 detection PASCAL VOC2012 segmentation
Metrics: test accuracy mini-batch saved wall-time speedup mAP mIoU
Baselines: No Removal Random Remove Small Loss & Rescale (InfoBatch) Conventional Early Stopping SB DIHCL EfficientTrain
Builds on: early stopping InfoBatch importance sampling Sharpness-Aware Minimization
Closest papers by meaning
- Instance-Conditional Timescales of Decay for Non-Stationary Learning (AAAI 2024 · similarity 0.84)
- ConsistentEE: A Consistent and Hardness-Guided Early Exiting Method for Accelerating Language Models Inference (AAAI 2024 · similarity 0.84)
- Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagation (ICML 2024 · similarity 0.84)
- How to Train Your Multi-Exit Model? Analyzing the Impact of Training Strategies (ICML 2025 · similarity 0.83)
- Learning the Optimal Stopping for Early Classification within Finite Horizons via Sequential Probability Ratio Test (ICLR 2025 · similarity 0.83)
- Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation is Wasteful (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.