Research map / Architectures and efficiency / RNNs and normalization
H3T: Efficient Integration of Memory Optimization and Parallelism for Large-scale Transformer Training
Yuzhong Wang, Xu Han, Weilin Zhao, Guoyang Zeng, Zhiyuan Liu, Maosong Sun
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace manual all-switches-on strategy with H3T DP solver | BMTrain · BERT-1.8B 8xRTX2080Ti bs128 | time per step (s): 7.1 -> 6.0 | Improved |
| Replace manual strategy with H3T DP solver | BMTrain · BERT-6B 8xRTX2080Ti bs8 | time per step (s): 22.1 -> 19.4 | Improved |
| Replace manual strategy with H3T DP solver when memory is plentiful | BMTrain · BERT-13B 64xA100 bs1024 | time per step (s): 7.8 -> 7.4 | Improved |
| Replace manual strategy with H3T DP solver | BMTrain · BERT-100B 64xA100 bs2048 | time per step (s): 61.8 -> 61.6 | No clear effect |
| Use greedy solver instead of DP solver (ablation) | H3T · BERT-6B 8xRTX2080Ti bs32 | time per step (s): 19.8 -> 20.5 | Hurt |
| Use greedy solver instead of DP solver (ablation) | H3T · BERT-1.8B 8xRTX2080Ti bs128 | time per step (s): 6.0 -> 5.8 | Improved |
| Use H3T instead of Megatron-DeepSpeed ZeRO-2 | BERT training · BERT-13B 8xA100 bs128 | time per step (s): 17.7 -> 10.0 | Improved |
| Use H3T instead of Megatron-DeepSpeed ZeRO-3 | BERT training · BERT-13B 8xA100 bs128 | time per step (s): 43.2 -> 10.0 | Improved |
| Use H3T instead of Megatron-DeepSpeed ZeRO-2 | BERT training · BERT-6B 8xA100 bs128 | time per step (s): 6.8 -> 5.5 | Improved |
| Larger model size widens H3T advantage over Megatron-DeepSpeed | H3T · BERT-6B vs BERT-13B, 8xA100 bs128 | speedup over ZeRO-2: 6.8 -> 5.5 (6B); 17.7 -> 10.0 (13B) | Improved |
| Use H3T instead of Megatron-DeepSpeed for minimal memory | BERT training · Minimal memory, all settings | GPU memory saving: 34.6% ~ 70.7% | Improved |
| Use H3T with random solver (ablation) | H3T · Actual-run settings | feasibility | Hurt |
Closest papers by meaning
- Trainable Transformer in Transformer (ICML 2024 · similarity 0.86)
- A General and Efficient Training for Transformer via Token Expansion (CVPR 2024 · similarity 0.86)
- StarTrail: Concentric Ring Sequence Parallelism for Efficient Near-Infinite-Context Transformer Model Training (NeurIPS 2025 · similarity 0.85)
- CODA: Rewriting Transformer Blocks as GEMM-Epilogue Programs (NeurIPS 2026 · similarity 0.85)
- Learning Rate Transfer for Hybrid Transformer-SSM Architectures (NeurIPS 2026 · similarity 0.85)
- IceFormer: Accelerated Inference with Long-Sequence Transformers on CPUs (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
