Research map / Large language models / Agents, code and math
Efficient Distributed MLLM Training with Cornstarch
Insu Jang, Runyu Lu, Nikhil Bansal, Ang Chen, Mosharaf Chowdhury
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| make pipeline partitioning frozen-status aware (ablation) | Cornstarch pipeline parallelism · MLLM-MMM | iteration time (s): 173.01 -> 70.37 | Improved |
| make pipeline partitioning frozen-status aware (ablation) | Cornstarch pipeline parallelism · MLLM-LLL | iteration time (s): 247.79 -> 143.76 | Improved |
| make pipeline partitioning frozen-aware when encoder and LLM sizes are extremely different (ablation) | Cornstarch pipeline parallelism · MLLM-LLS and MLLM-SSL | iteration time (s): 66.20 -> 66.14 | No clear effect |
| add inter-GPU token balancing only to causal CP (ablation) | context parallelism · Attention layer, 64k, LLM-L | time (ms): 568.18 -> 610.67 | Hurt |
| add intra-GPU balancing only (ablation) | context parallelism · Attention layer, 64k, LLM-S | time (ms): 243.44 -> 225.73 | Improved |
| combine inter-GPU and intra-GPU balancing (ablation) | context parallelism · Attention layer, 64k, LLM-S | time (ms): 243.44 -> 204.95 | Improved |
| use Cornstarch all-gather CP instead of DCP ring CP at short sequence length | context parallelism · Lambda attention, 32k | attention layer time (ms): 46.36 -> 177.41 | Hurt |
| use Cornstarch all-gather CP instead of DCP ring CP at long sequence length | context parallelism · Lambda attention, 256k | attention layer time (ms): 1545.83 -> 1193.32 | Improved |
| use heuristic token assignment instead of DCP planning | end-to-end training · Causal, 32k | iteration time (s): 26.5 -> 5.7 | Improved |
| use Cornstarch instead of FSDP and Megatron* | MLLM training · 24 A40 GPUs, various MLLM configs | speedup: 3.36× vs FSDP, 1.62× vs Megatron* | Improved |
Closest papers by meaning
- Rethinking Memory and Communication Costs for Efficient Data Parallel Training of Large Language Models (NeurIPS 2024 · similarity 0.89)
- EDiT: A Local-SGD-Based Efficient Distributed Training Method for Large Language Models (ICLR 2025 · similarity 0.88)
- HALoS: Hierarchical Asynchronous Local SGD over Slow Networks for Geo-Distributed Large Language Model Training (ICML 2025 · similarity 0.88)
- NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints (NeurIPS 2025 · similarity 0.88)
- Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (ICLR 2024 · similarity 0.88)
- Heterogeneous Parallelism for Multimodal Large Language Model Training (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
