Research map / Architectures and efficiency / Pruning and efficient tuning
InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InfoBatch, an unbiased dynamic data pruning method that randomly prunes low-loss samples each epoch and rescales remaining low-loss gradients, with full-data annealing at the end, to achieve lossless training speedups across classification, segmentation, MAE pretraining, diffusion, and LLaMA instruction tuning.
Problem
Reducing training cost of deep networks on large-scale datasets; existing static and dynamic data pruning methods introduce gradient expectation bias, have high scoring/sorting overhead, and rarely achieve lossless accuracy at meaningful pruning ratios.
Approach
Randomly prune a fraction r of samples whose loss is below the current mean loss each epoch (soft pruning, O(1) per sample via mean comparison), rescale the gradients of the remaining below-mean samples by 1/(1-r) to preserve the expected gradient, and train on the full dataset in the final epochs (annealing) to reduce variance.
Key results
- On CIFAR-10 with ResNet-18 at 30% prune ratio, InfoBatch reaches 95.6% (lossless vs. 95.6% full data), while the best static baselines drop by up to 1 point and random dynamic pruning drops 0.8-2.6 points.
- On ImageNet-1K with ResNet-50 at 40% prune ratio, InfoBatch achieves 76.5% vs 76.4% full data, with total cost 84 node-hours vs 140 for full training and 0.0028 h overhead vs 0.03 h for UCB.
- On ADE20K with UperNet, InfoBatch matches baseline mIoU (41.12% vs 40.7%) using 60% of iterations; on Latent Diffusion FFHQ it saves ~27% cost with FID 7.70 vs 7.83.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use InfoBatch instead of dynamic random pruning at 30% pruning | ResNet-18 · CIFAR-100 | accuracy (%): 77.3 -> 78.2 | Helped |
| Add expectation rescaling to soft pruning (ablation) | InfoBatch, ResNet-18 · CIFAR-100 | accuracy (%): 77.5 -> 78.1 | Helped |
| Add annealing to soft pruning and rescaling (ablation) | InfoBatch, ResNet-18 · CIFAR-100 | accuracy (%): 78.1 -> 78.2 | Helped |
| Add rescaling to soft pruning and annealing (ablation) | InfoBatch, ResNet-18 · CIFAR-100 | accuracy (%): 77.7 -> 78.2 | Helped |
| Add annealing after rescaling (ablation) | InfoBatch · CIFAR-100 | accuracy (%) | Mixed |
| Replace low-loss pruning with high-loss pruning (ablation) | InfoBatch, ResNet-50 · CIFAR-100 | accuracy (%): 80.6 -> 80.5 | Hurt |
| Replace low-loss pruning with random pruning at 33% prune (ablation) | InfoBatch, ResNet-50 · CIFAR-100 | accuracy (%): 80.6 -> 80.5 | Hurt |
| Increase pruning probability to r≥0.6 (ablation) | InfoBatch · CIFAR-100 | accuracy | Hurt |
| Use fewer annealing epochs by increasing δ (ablation) | InfoBatch · CIFAR-100 | accuracy | Hurt |
| Use InfoBatch instead of full-data training | ResNet-50 · ImageNet-1K | accuracy (%): 76.4 -> 76.5 | No clear effect |
| Use InfoBatch instead of full-data training | UperNet, ResNet-50 · ADE20K | mIoU (%): 40.7 -> 41.12 | Helped |
| Use InfoBatch for diffusion-model training | Latent Diffusion · FFHQ | FID: 7.83 -> 7.70 | Helped |
Surprising findings
- Hard-pruning low-loss samples can be less effective than soft pruning with rescaling; the paper argues unbiased hard pruning would imply the pruned set itself is a valid compression, contradicting common assumptions.
- Training the full dataset for the final ~12.5% of epochs (annealing) is needed for stable lossless results; rescaling alone gives higher variance and lower mean accuracy.
Limitations
- Relies on multi-epoch training schemes; may need adaptation for limited-epoch training such as GPT-3 or ViT-22B.
- Approximation holds only when pruning ratio is not too high and learning rate is not too large; r >= 0.6 degrades performance.
- Loss-based scoring for segmentation requires a heuristic (weighted sample-level sum of two losses).
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-1K ADE20K FFHQ Tiny-ImageNet Alpaca
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-1K ADE20K FFHQ BBH DROP MMLU Human-Eval
Metrics: top-1 accuracy mIoU FID saved overall cost (%) wall-clock time overhead (h)
Baselines: Random CD Herding K-Center EL2N Forgetting GraNd Glister UCB epsilon-greedy Random* GC
Builds on: UCB dynamic pruning epsilon-greedy dynamic pruning EL2N LARS LAMB
Closest papers by meaning
- Exploring Learning Complexity for Efficient Downstream Dataset Pruning (ICLR 2025 · similarity 0.88)
- Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty (ICML 2025 · similarity 0.87)
- Data Pruning by Information Maximization (ICLR 2025 · similarity 0.86)
- Effective pruning of web-scale datasets based on complexity of concept clusters (ICLR 2024 · similarity 0.85)
- Distilling the Knowledge in Data Pruning (ICML 2025 · similarity 0.85)
- DeepSpeed Data Efficiency: Improving Deep Learning Model Quality and Training Efficiency via Efficient Data Sampling and Routing (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.