Research map / Architectures and efficiency / Pruning and efficient tuning
Spanning Training Progress: Temporal Dual-Depth Scoring (TDDS) for Enhanced Dataset Pruning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Temporal Dual-Depth Scoring (TDDS), a dataset pruning method that scores samples by the temporal variance of projected per-sample loss changes over sliding windows, with EMA aggregation, to build coresets that generalize across pruning rates and architectures.
Problem
Snapshot-based dataset pruning criteria yield coresets that fluctuate with the chosen training epoch and generalize poorly across pruning rates and architectures, while existing training-dynamics methods average dynamics and overlook well-generalized samples.
Approach
A dual-depth temporal score: the inner depth estimates each sample's per-epoch contribution as a KL-based loss difference projected onto the accumulated gradient direction; the outer depth measures the variance of these contributions over sliding windows (EMA-aggregated) instead of averaging, and the coreset is trained with importance-weighted loss.
Key results
- On CIFAR-100 with ResNet-18 under Strategy-E at 90% pruning, TDDS reaches 54.51% accuracy, surpassing Random by 7.83% and the best baseline by more than 12.69%.
- On ImageNet-1K with Swin-T at 50% pruning, TDDS achieves 78.33%, exceeding Forgetting, Moderate and Dyn-Unc by 4.01%, 3.35% and 0.69%.
- Under Strategy-P at 30% on CIFAR-10, TDDS achieves 93.92%, outperforming CCS by 0.92%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use TDDS instead of the strongest listed baseline | ResNet-18 dataset pruning · CIFAR-100, Strategy-E, 90% pruning | accuracy: 46.68 -> 54.51 | Helped |
| use TDDS instead of CCS | ResNet-18 dataset pruning · CIFAR-100, Strategy-P, 70% pruning | accuracy: 68.92 ±0.12 -> 69.46 ±0.10 | Helped |
| add the outer depth to EL2N (ablation) | CIFAR-100 pruning with ResNet-18 · 30% pruning | accuracy: 76.89 -> 77.47 [+ 0.58] | Helped |
| add the outer depth to EL2N (ablation) | CIFAR-100 pruning with ResNet-18 · 50% pruning | accuracy: 67.57 -> 73.93 [+ 6.36] | Helped |
| add the outer depth to EL2N (ablation) | CIFAR-100 pruning with ResNet-18 · 70% pruning | accuracy: 36.45 -> 67.04 [+30.59] | Helped |
| add the outer depth to EL2N (ablation) | CIFAR-100 pruning with ResNet-18 · 80% pruning | accuracy: 17.31 -> 58.21 [+40.90] | Helped |
| add the outer depth to EL2N (ablation) | CIFAR-100 pruning with ResNet-18 · 90% pruning | accuracy: 9.10 -> 40.36 [+31.26] | Helped |
| use gradient magnitudes instead of gradients to estimate contribution (ablation) | TDDS · CIFAR pruning | accuracy | Helped |
| use KL loss instead of CE loss for loss difference (ablation) | TDDS · CIFAR pruning at 30% pruning | accuracy | Hurt |
| use KL loss instead of CE loss for loss difference (ablation) | TDDS · CIFAR pruning at pruning rates other than 30% | accuracy: +15.97% | Helped |
| weight coreset samples by their importance scores during training (ablation) | TDDS · CIFAR pruning at 90% pruning | accuracy: 6.55% | Helped |
| use EMA instead of simple averaging (ablation) | TDDS · CIFAR-100 | accuracy | Helped |
Surprising findings
- Averaging training dynamics (as in Forgetting, AUM, and EL2N-style aggregation) can overlook well-generalized samples, and the variance-based outer depth adds large gains at high pruning rates (e.g., +30.59% at 70% pruning on CIFAR-100 when added to EL2N).
Limitations
- Strategy-P experiments only run at pruning rates 0.3, 0.5, 0.7 due to training cost.
- Hyperparameters such as range T, window size K, and batch size are tuned per pruning rate, with optimal settings varying across settings.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-1K
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-1K
Metrics: top-1 accuracy
Baselines: Random Entropy Forgetting EL2N AUM Moderate CCS Dyn-Unc
Builds on: Forgetting AUM EL2N Dyn-Unc Moderate coreset
Closest papers by meaning
- Exploring Learning Complexity for Efficient Downstream Dataset Pruning (ICLR 2025 · similarity 0.88)
- $\mathbb{D}^2$ Pruning: Message Passing for Balancing Diversity & Difficulty in Data Pruning (ICLR 2024 · similarity 0.87)
- Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty (ICML 2025 · similarity 0.87)
- Distilling the Knowledge in Data Pruning (ICML 2025 · similarity 0.87)
- GDeR: Safeguarding Efficiency, Balancing, and Robustness via Prototypical Graph Pruning (NeurIPS 2024 · similarity 0.86)
- Effective pruning of web-scale datasets based on complexity of concept clusters (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.