Research map / Architectures and efficiency / RNNs and normalization
Deep Learning on a Data Diet: Finding Important Examples Early in Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| prune lowest-EL2N 50% of training data using scores early in training | ResNet18 · CIFAR-10 | test accuracy | No clear effect |
| prune 25% of CIFAR-100 by EL2N early scores | ResNet18 · CIFAR-100 | test accuracy: 1% drop (GraNd/at 25%) / no loss with EL2N | Hurt |
| average scores over ten initializations instead of a single network (ablation) | EL2N/GraNd · CIFAR-10 | pruning test accuracy | Helped |
| prune using GraNd scores at initialization versus random | GraNd · CIFAR-10/100, CINIC-10 | test accuracy | Helped |
| prune using EL2N scores at initialization | EL2N · CIFAR-10/100, CINIC-10 | test accuracy | No clear effect |
| compute EL2N scores after a few epochs instead of at initialization (ablation) | EL2N · CIFAR-10 | test accuracy | Helped |
| prune at extreme pruning levels (keep only highest scores) | EL2N/GraNd · CIFAR-10 | test accuracy | Hurt |
| exclude the very highest-scoring examples (sliding window) (ablation) | EL2N pruning · CIFAR-10 | test accuracy | Helped |
| increase label corruption (ablation) | EL2N sliding window · CIFAR-10 with noisy labels | optimal number of top examples excluded | Mixed |
| transfer EL2N scores across architectures/hyperparameters | EL2N · CIFAR-10 | pruning test accuracy | No clear effect |
Closest papers by meaning
- Deep Learning Through the Lens of Example Difficulty (NeurIPS 2021 · similarity 0.89)
- Characterizing Datapoints via Second-Split Forgetting (NeurIPS 2022 · similarity 0.88)
- To Grok or not to Grok: Disentangling Generalization and Memorization on Corrupted Algorithmic Datasets (ICLR 2024 · similarity 0.86)
- PEAKS: Selecting Key Training Examples Incrementally via Prediction Error Anchored by Kernel Similarity (ICML 2025 · similarity 0.86)
- Changing the Training Data Distribution to Reduce Simplicity Bias Improves In-distribution Generalization (NeurIPS 2024 · similarity 0.86)
- From Linear to Nonlinear: Provable Weak-to-Strong Generalization through Feature Learning (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.