Research map / Architectures and efficiency / RNNs and normalization
DeepSpeed Data Efficiency: Improving Deep Learning Model Quality and Training Efficiency via Efficient Data Sampling and Routing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents DeepSpeed Data Efficiency, combining a general curriculum learning library for data sampling with random layerwise token dropping (random-LTD) for data routing to cut GPT-3 1.3B and BERT-large pretraining cost by up to 12.5x and 2x respectively.
Problem
Foundation model pretraining cost grows with both model and data scale, yet data efficiency is underexplored and existing curriculum learning and token-dropping methods are hard to implement at scale or lack generality for training.
Approach
Combine a scalable map-reduce curriculum learning library (with seven difficulty metrics and a general sampler) with random layerwise token dropping, which independently drops random tokens at each middle layer without importance scores or special-token rules, and apply token-based LR decay to both.
Key results
- GPT-3 1.3B with CL+random-LTD at 50% data (150B tokens) matches the 100% baseline average 0-shot accuracy (42.8 vs 42.5), a 2x data and 2x time saving.
- Up to 12.5x data/time/cost saving (from $46.3K to $3.7K on Azure) while retaining 95% of baseline zero-shot quality at 1% to 3B tokens.
- BERT-large with random-LTD at 2x less data (524B tokens) reaches GLUE 87.32 vs baseline 86.61 at full data budget comparisons, surpassing TokenBypass's 1.33x data saving.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply seqtru_voc curriculum learning with random-LTD | GPT-3 1.3B · 300B-token pretraining | 0-shot accuracy: 42.5 -> 43.8 | Helped |
| Apply seqtru_voc curriculum learning with random-LTD | GPT-3 1.3B · 150B-token pretraining | 10-shot accuracy: 42.7 -> 44.0 | Helped |
| Apply seqtru_voc curriculum learning with random-LTD | BERT-large · 524B-token pretraining | GLUE score: 86.61±0.5 -> 87.44±0.46 | Helped |
| Apply random-LTD | BERT-large · 1049B-token pretraining | GLUE score: 87.29±0.53 -> 88.17±0.48 | Helped |
| Add random-LTD to seqtru_voc curriculum learning (ablation) | BERT-large · 1049B-token pretraining | GLUE score: 87.60±0.34 -> 87.69±0.32; 88.17±0.48 | Mixed |
| Apply CL_seqres | GPT-2 · PTB finetuning | PPL median/std: 16.077±0.028 -> 15.818±0.032 | Helped |
| Apply random-LTD | GPT-2 · PTB finetuning | PPL median/std: 16.077±0.028 -> 15.948±0.040 | Helped |
| Use CL_seqtru_voc instead of CL_seqtru (ablation) | GPT-3 1.3B · 300B-token pretraining | 0-shot accuracy: 43.4 -> 43.6 | Helped |
| Use CL_voc (ablation) | GPT-3 1.3B · 300B-token pretraining | 0-shot accuracy: 42.5 -> 42.3 | Hurt |
| Use CL_seqtru_voc instead of CL_seqtru (ablation) | BERT-large · 1049B-token pretraining | GLUE score: 87.31±0.57 -> 87.60±0.34 | Helped |
| Use CL_seqreo_voc (ablation) | BERT-large · 1049B-token pretraining | GLUE score: 87.29±0.53 -> 87.06±0.52 | Hurt |
| Use the monotonic sequence length growth schedule instead of constant token dropping (ablation) | random-LTD · Training | model quality | Helped |
Surprising findings
- Random-LTD with MSLG achieves 8.26 validation loss vs 9.62 for TokenBypass under the same token saving, suggesting purely random dropping outperforms importance-score-based dropping.
- Composing CL and random-LTD at 100% BERT data gave worse GLUE than random-LTD alone (87.69 vs 88.17).
Limitations
- Many results are single-seed for GPT-3 pretraining, and 0/10-shot evaluations give no variance estimates.
- Curriculum-based slowdown in early training and token-based LR changes require careful tuning, though a low-cost tuning strategy is proposed.
- Cost claims rely on measured time on rented Azure hardware and are specific to the setup.
Experimental setup
Datasets: The Pile PTB GLUE ImageNet CIFAR
Benchmarks: HellaSwag LAMBADA TriviaQA WebQs Winogrande PIQA ARC ANLI OpenBookQA RACE-h BoolQ Copa
Metrics: 0-shot accuracy 10-shot accuracy GLUE score perplexity training time/cost
Baselines: baseline uniform sampling training TokenBypass SLW (sequence length warmup) OpenAI GPT-3 1.3B
Builds on: curriculum learning TokenBypass Transformer DeepSpeed
Closest papers by meaning
- Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty (ICML 2025 · similarity 0.86)
- Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation is Wasteful (NeurIPS 2025 · similarity 0.86)
- Model Reprogramming: Resource-Efficient Cross-Domain Machine Learning (AAAI 2024 · similarity 0.85)
- Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.85)
- Efficiency for Free: Ideal Data Are Transportable Representations (NeurIPS 2024 · similarity 0.85)
- SpeedLoader: An I/O efficient scheme for heterogeneous and distributed LLM operation (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.