Research map / Architectures and efficiency / Pruning and efficient tuning
Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DUAL, an early-training dataset pruning score combining example difficulty (1 - mean prediction) and prediction standard deviation, plus a pruning-ratio-adaptive Beta sampling scheme to handle extreme pruning ratios.
Problem
Dataset pruning methods typically require fully training on the whole dataset for many epochs to compute sample scores, making the pruning pipeline more expensive than training itself; accuracy also drops at extreme pruning ratios with score-only selection.
Approach
Score samples at an early training epoch (e.g., 30-60) by the product of difficulty (1 minus the windowed mean target probability) and windowed prediction standard deviation (uncertainty), averaged over windows; then sample subsets with a Beta-PDF selection probability whose mean shifts toward easy samples as the pruning ratio grows.
Key results
- On ImageNet-1k (ResNet-34), DUAL+beta sampling achieves 60.0% test accuracy at 90% pruning, versus 57.3% for the next best (CCS).
- On CIFAR-10 at 90% pruning, DUAL+beta sampling reaches 87.09% vs 85.92% for CCS; the DUAL score alone computed at epoch 30 reduces total time to about 15% of full training on CIFAR datasets.
- On CIFAR-100 with 20% symmetric label noise, DUAL+beta sampling reaches 54.04% at 90% pruning vs 48.00% for Dyn-Unc.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add pruning-ratio-adaptive β sampling to DUAL (ablation) | DUAL on CIFAR-10 at 90% pruning · CIFAR-10, 90% pruning | test accuracy: 54.95 -> 87.09 | Helped |
| add pruning-ratio-adaptive β sampling to DUAL (ablation) | DUAL on CIFAR-100 at 90% pruning · CIFAR-100, 90% pruning | test accuracy: 34.28 -> 54.54 | Helped |
| add pruning-ratio-adaptive β sampling to DUAL (ablation) | DUAL on ImageNet-1K at 90% pruning · ImageNet-1K, 90% pruning | test accuracy: 53.1 -> 60.0 | Helped |
| apply β sampling instead of thresholding (ablation) | EL2N on CIFAR-10 at 90% pruning · CIFAR-10, 90% pruning | test accuracy: 38.74 -> 87.00 (+48.26) | Helped |
| apply β sampling instead of thresholding (ablation) | EL2N on CIFAR-100 at 90% pruning · CIFAR-100, 90% pruning | test accuracy: 8.89 -> 53.97 (+45.08) | Helped |
| apply β sampling instead of thresholding (ablation) | Random pruning on CIFAR-10 at 90% pruning · CIFAR-10, 90% pruning | test accuracy: 83.74 -> 83.31 (-0.43) | Hurt |
| apply β sampling instead of thresholding (ablation) | Random pruning on CIFAR-100 at 90% pruning · CIFAR-100, 90% pruning | test accuracy: 45.09 -> 51.76 (+6.67) | Helped |
| vary score-computation epoch T from 20 to 200 (ablation) | DUAL on CIFAR-100 · CIFAR-100 | test accuracy | Mixed |
| vary sliding-window length J (ablation) | DUAL · CIFAR-100 | test accuracy | No clear effect |
| vary dataset hyperparameter c_D from 3 to 6 (ablation) | DUAL on CIFAR-100 · CIFAR-100 | test accuracy | No clear effect |
Surprising findings
- Score-only DUAL selection degrades sharply at 90% pruning on CIFAR-10 (54.95%), while combining with Beta sampling gives 87.09%.
- Dyn-Unc performs worse than random pruning on ImageNet-1k at all pruning ratios when scored only over 90 epochs.
Limitations
- Score-based selection alone suffers accuracy drops at extreme pruning ratios, requiring the Beta sampling add-on.
- Hyperparameters (T, J, c_D) are dataset-specific and chosen per dataset; c_D sensitivity is only partly explored.
- Evaluated only on supervised image classification; the authors note extension to unsupervised settings as future work.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-1k Tiny-ImageNet CIFAR-10-LT CIFAR-100-LT
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-1k Tiny-ImageNet
Metrics: test accuracy total time cost pruned mislabeled data ratio Spearman rank correlation
Baselines: Random Entropy Forgetting EL2N AUM Moderate Dyn-Unc TDDS CCS D2
Builds on: Dyn-Unc BOSS TDDS CCS D2
Closest papers by meaning
- Exploring Learning Complexity for Efficient Downstream Dataset Pruning (ICLR 2025 · similarity 0.92)
- Label-Efficient Dataset Pruning via Semi-Supervised Pseudo-Labeling (NeurIPS 2026 · similarity 0.90)
- DRoP: Distributionally Robust Data Pruning (ICLR 2025 · similarity 0.89)
- $\mathbb{D}^2$ Pruning: Message Passing for Balancing Diversity & Difficulty in Data Pruning (ICLR 2024 · similarity 0.89)
- Scale Efficient Training for Large Datasets (CVPR 2025 · similarity 0.88)
- GDeR: Safeguarding Efficiency, Balancing, and Robustness via Prototypical Graph Pruning (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.