Research map / Large language models / Language, safety and interpretability
Efficient Pre-Training of LLMs via Topology-Aware Communication Alignment on More Than 9600 GPUs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Arnold is a topology-aware GPU scheduling system that aligns LLM pre-training communication groups with data-center network topology, using an MIP-based placement with DP/PP trade-off weights and a queue-reservation policy.
Problem
Inefficient placement of LLM pre-training jobs across a multi-tier fat-tree/CLOS GPU cluster causes cross-switch communication and bandwidth contention, hurting end-to-end training throughput at scale.
Approach
Model each LLM pre-training job as a communication matrix of DP and PP groups and minimize the weighted maximum spread of these groups across minipods via a bin-packing-like MIP, with affinity weights derived from pre-characterized DP-vs-PP placement gains, plus a reservation-based queue policy for imminent large jobs.
Key results
- Simulation: the proposed algorithm reduces maximum spread of communication groups by up to 1.67x versus the best baseline, and about 1.2x on average.
- Production full-scale run with more than 9600 GPUs improves end-to-end throughput by 10.6% over MegaScale; the 208-GPU run gives 5.7%.
- Characterization: performance drops up to 17% for collective operations and 70% for P2P as communication spans more minipods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use topology-aware communication alignment scheduling instead of the best baseline | Arnold · Simulation benchmark settings (i–iii) | weighted maximum communication-group spread: up to 1.67x | Helped |
| use topology-aware communication alignment scheduling instead of MegaScale | LLM pre-training on 208 GPUs · Production cluster, 208 GPUs | average throughput: 5.7% | Helped |
| use topology-aware communication alignment scheduling instead of MegaScale | LLM pre-training on 9600+ GPUs · Production cluster, 9600+ GPUs | average throughput: 10.6% | Helped |
| prioritize PP-group placement rather than DP-aligned placement (ablation) | dense model · 96 GPUs across 2 minipods | throughput: 2.3% | Helped |
| prioritize PP-group placement rather than DP-aligned placement (ablation) | MoE model · 96 GPUs across 2 minipods | throughput: 1.8% | Helped |
| prioritize DP-group placement (ablation) | dense model · 96 GPUs across 2 minipods | throughput | No clear effect |
| scale the model size by adding more layers (ablation) | LLM training with optimized alignment · Model sizes scaled up | performance improvement | Helped |
| vary node placement within a single minipod (ablation) | dense 24B model · Intra-minipod topology sensitivity | training performance variation: 0.3% | No clear effect |
| prioritize DP-group placement (ablation) | dense 7b model on L20 GPUs · L20 cluster | speedup: 1.4% | Helped |
| prioritize PP-group placement (ablation) | dense 14b model on L20 GPUs · L20 cluster | speedup: 0.5% | Helped |
| increase GPU SM allocation to communication using the NCCL variable (ablation) | Arnold communication and computation kernels · Full-scale production experiment | computation-kernel performance | Hurt |
| use GBM rather than DNN for JCT prediction (ablation) | JCT predictor · JCT prediction | RMSE: GBM: RMSE 1.61. DNN: RMSE: 2.12. | Helped |
Surprising findings
- Topology-aligned placement can speed up communication kernels yet slow down some computation kernels (e.g., GEMM) due to contention between GPU streams.
- Intra-minipod node placement has negligible effect on training performance (max variation 0.3%), so cross-minipod spread is what matters.
Limitations
- Failure recovery: optimal placement changes on hardware failure and re-solving the MIP and migrating is expensive.
- Results derived from in-house CLOS-based H800 data center; generalization to other topologies (e.g., newer LLM-specific networks) is not validated.
- Production experiments use hidden model and GPU counts and a single proprietary system; reproducibility is limited.
Experimental setup
Datasets: historical job traces from production cluster (4-month trace for JCT predictor) NCCL benchmark measurements
Benchmarks: trace-driven simulation on three topology settings 96-GPU down-scaled production run 208-GPU production run 9600+ GPU production run
Metrics: weighted sum of maximum spread of communication groups scheduling latency BusBw end-to-end training throughput speedup RMSE for JCT prediction
Baselines: Best-fit Random-fit GPU-packing Topo-aware MegaScale
Builds on: Megatron MegaScale Kubernetes ZeRO
Closest papers by meaning
- ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training (NeurIPS 2025 · similarity 0.87)
- Revisiting Parameter Server in LLM Post-Training (ICLR 2026 · similarity 0.87)
- SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training (NeurIPS 2024 · similarity 0.87)
- Communication-Efficient LLM Adaptation over Decentralized GPU Meshes (NeurIPS 2026 · similarity 0.86)
- Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs (ICML 2025 · similarity 0.86)
- ReCoVer: Resilient LLM Pre-Training System via Fault-Tolerant Collective and Versatile Workload (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.