Research map / Architectures and efficiency / Pruning and efficient tuning
Repeated Random Sampling for Minimizing the Time-to-Accuracy of Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RS2, which resamples a fresh random subset of training data each epoch (with or without replacement) and shows it beats 30 data pruning and distillation baselines on time-to-accuracy across CIFAR10/100, ImageNet and GPT2 pretraining.
Problem
Data pruning, coreset selection and dataset distillation reduce training cost but their subset selection overhead is large, and score-based methods often underperform random sampling at high compression, so time-to-accuracy is not reduced.
Approach
Instead of selecting a static subset or running costly selection, draw a fresh uniformly random subset of size r·N each epoch (RS2), using a learning rate schedule adjusted for the reduced number of SGD steps; a without-replacement variant is equivalent to early stopping over a random permutation.
Key results
- On CIFAR10 with 5% of data per epoch, RS2 without replacement reaches 87.1% accuracy versus 65.7% for the next closest baseline.
- On ImageNet with r=1%, RS2 reaches 46.96% accuracy versus 18.1% for the next best baseline; at r=10% RS2 reaches 66% and reaches it about 9x faster than full-data training.
- RS2 with r=10% trains GPT2 with accuracy/perplexity within about 2 points/5 points of full-dataset training at 10x lower cost per the reported cost table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| repeatedly sample random subsets instead of using a static subset | ResNet-18 training · CIFAR10, 5% | accuracy: 65.7$\pm$1.2 -> 87.1$\pm$0.8 | Helped |
| repeatedly sample random subsets instead of using a static subset | ResNet-18 training · ImageNet, 1% | accuracy: 18.1\% -> 46.96\% | Helped |
| use the full-dataset learning-rate schedule without faster decay (ablation) | RS2 on CIFAR10 · CIFAR10, 10% | accuracy: 83.9\% -> 91.7\% | Hurt |
| sample random subsets repeatedly with replacement rather than without replacement (ablation) | RS2 · CIFAR10, 10% | accuracy: 91.7\% -> 89.7\% | Hurt |
| resample subsets according to example-importance scores rather than uniformly at random | per-round data pruning · CIFAR10, 5% | accuracy: 86.2$\pm$2.1 -> 87.1$\pm$0.8 | Hurt |
| use stratified sampling (ablation) | RS2 with replacement · ImageNet, 10% | accuracy: 64.87$\pm$0.10 -> 64.92$\pm$0.10 | No clear effect |
| sample subsets without replacement rather than with replacement (ablation) | RS2 · ImageNet, 10% | accuracy: 64.87$\pm$0.10 -> 66.0$\pm$0.18 | Helped |
| replace static data pruning or distillation subsets with repeated random sampling | ConvNet training · Tiny ImageNet, 0.2% | accuracy: 8.8$\pm$0.3 -> 23.5$\pm$0.2 | Helped |
| repeatedly sample random text subsets rather than use a static random subset | GPT2 pretraining · OpenWebText, LAMBADA | accuracy: 44.63 -> 45.29 | Helped |
| repeatedly sample random subsets rather than use a static random subset | GPT2 pretraining · OpenWebText, LAMBADA | perplexity: 41.99 -> 40.51 | Helped |
| train with RS2 at 30% rather than on the full dataset | GPT2 pretraining · OpenWebText, LAMBADA | accuracy: 46.61 -> 45.29 | Hurt |
| use repeated random sampling without replacement (ablation) | ResNet-18 training · CIFAR10, 30% | accuracy: 94.5$\pm$0.1 -> 94.3$\pm$0.2 | No clear effect |
Surprising findings
- Random sampling repeated each epoch outperforms 22 data pruning and 8 dataset distillation methods in the high compression regime.
- In extreme compression (e.g., r=0.1% on ImageNet), RS2 is inferior to some existing static pruning methods.
Limitations
- RS2 may not outperform existing methods when the full dataset is unlabeled or the goal is labeling cost reduction (active learning regime).
- RS2 is inferior to static pruning methods at extreme compression ratios (e.g., r=0.1% on ImageNet).
- Training on a subset yields fewer SGD iterations, leaving a gap to full-dataset accuracy.
Experimental setup
Datasets: CIFAR10 CIFAR100 ImageNet30 ImageNet Tiny ImageNet OpenWebText
Benchmarks: CIFAR10 CIFAR100 ImageNet30 ImageNet Tiny ImageNet LAMBADA WikiText103
Metrics: top-1 accuracy time-to-accuracy runtime perplexity relative accuracy drop under label noise
Baselines: Random static subset Forgetting GraNd CAL Craig Glister GraphCut Active learning (Conf/LL/Margin) SP-Easy Trajectory Matching DM CAFE
Builds on: cosine annealing Nesterov accelerated gradient EL2N Trajectory Matching
Closest papers by meaning
- Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty (ICML 2025 · similarity 0.88)
- Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets? (NeurIPS 2026 · similarity 0.87)
- $\mathbb{D}^2$ Pruning: Message Passing for Balancing Diversity & Difficulty in Data Pruning (ICLR 2024 · similarity 0.86)
- Embarrassingly Simple Dataset Distillation (ICLR 2024 · similarity 0.86)
- Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation (ICLR 2025 · similarity 0.86)
- Distilling the Knowledge in Data Pruning (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.