Research map / Architectures and efficiency / Pruning and efficient tuning
Scale Efficient Training for Large Datasets
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SeTa, a dynamic sample pruning method that randomly downsamples, clusters samples by loss, and slides a window over easy-to-hard clusters with partial annealing to cut training cost up to 50% without loss.
Problem
Large-scale training is inefficient because datasets contain low-value samples (redundant, overly hard, and easy samples that add little), and existing loss-based dynamic pruning ignores redundant and overly hard samples.
Approach
Combine random downsampling for redundancy removal, loss-guided k-means clustering into difficulty groups, and a cyclic sliding-window curriculum over those groups (easy-to-hard), followed by partial annealing, to dynamically prune samples in a plug-and-play way.
Key results
- On ToCa, SeTa reaches 71.5 overall CIDEr vs 70.5 full-data baseline while pruning 31.7% of data.
- On CIFAR100 with ResNet18, SeTa achieves 79.4 at 30% pruning (baseline 79.3) and 77.7 at 70% pruning (-1.6).
- On MJ+ST, SeTa prunes 71.0% of data with IIIT5k accuracy 95.8% vs 96.1% baseline, where InfoBatch maxes at 50.3% pruning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| prune 30% of training data with SeTa | ResNet18 · CIFAR10 | accuracy: 95.6 -> 95.7 | Helped |
| prune 70% of training data with SeTa | ResNet18 · CIFAR10 | accuracy: 95.6 -> 95.0 | Hurt |
| prune 30% of training data with SeTa | ResNet18 · CIFAR100 | accuracy: 79.3 -> 79.4 | Helped |
| prune 31.7% of training data with SeTa | ViECap · ToCa, NoCaps Val Overall | CIDEr: 70.5 -> 71.5 | Helped |
| prune 41.8% of training data with SeTa | ViECap · ToCa, NoCaps Val Overall | CIDEr: 70.5 -> 69.3 | Hurt |
| prune 40.0% of training data with SeTa | CapDec · SS1M, COCO | CIDEr: 42.4 -> 45.1 | Helped |
| prune 71.0% of training data with SeTa | ABINet · ST+MJ, IC15 | accuracy: 85.4 -> 84.9 | Hurt |
| prune 40.4% of training data with SeTa | Ada-MVS · WHU-MVS | MAE (m): 0.1185 -> 0.1184 | Helped |
| prune 33.6% of training data with SeTa | GeoDTR · CVACT | R@1: 86.21 -> 86.24 | Helped |
| prune 25.9% of training data with SeTa | LLaMA-7B · Alpaca, instruction tuning | Avg.: 26.3 -> 27.3 | Helped |
| prune 60.5% of training data with SeTa | CaLa · CIRR | Recall@K Avg: 73.4 -> 73.5 | Helped |
| prune 30.7% of training data with SeTa | ViECap · COCO, image caption | CIDEr: 91.5 -> 92.8 | Helped |
Surprising findings
- SeTa outperforms full-data training on ToCa (71.5 vs 70.5 CIDEr) while using only 68.3% of the data.
- Dynamic random sampling outperforms the InfoBatch-style baseline at some pruning ratios on CIFAR100 in the reported tables.
Limitations
- Gains on some tasks are marginal or negative at high pruning rates (e.g., MJ+ST at 71% pruning, COCO caption and CVACT degradation at higher ratios).
- Efficiency is reported primarily as pruning ratio rather than measured wall-clock time.
Experimental setup
Datasets: ToCa SS1M MJ+ST ImageNet-1K COCO CIFAR10 CIFAR100 RefCOCO CIRR WHU-MVS CVACT Alpaca
Benchmarks: CIFAR10 CIFAR100 ToCa SS1M MJ+ST ImageNet-1K COCO NoCaps Flickr30k WHU-MVS CVACT CIRR
Metrics: accuracy CIDEr BLEU-4 Recall@K MAE pruned ratio Avg score
Baselines: InfoBatch UCB Dynamic Random Static Random epsilon-greedy Dyn-Unc DQ
Builds on: InfoBatch UCB EfficientTrain++
Closest papers by meaning
- Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty (ICML 2025 · similarity 0.88)
- Exploring Learning Complexity for Efficient Downstream Dataset Pruning (ICLR 2025 · similarity 0.87)
- Label-Efficient Dataset Pruning via Semi-Supervised Pseudo-Labeling (NeurIPS 2026 · similarity 0.85)
- Repeated Random Sampling for Minimizing the Time-to-Accuracy of Learning (ICLR 2024 · similarity 0.85)
- Effective pruning of web-scale datasets based on complexity of concept clusters (ICLR 2024 · similarity 0.85)
- $\mathbb{D}^2$ Pruning: Message Passing for Balancing Diversity & Difficulty in Data Pruning (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.