AI Paper Atlas

Research map / Large language models / Agents, code and math

Efficient Distributed MLLM Training with Cornstarch

Insu Jang, Runyu Lu, Nikhil Bansal, Ang Chen, Mosharaf Chowdhury

ICML 2026 · Large language models › Agents, code and math · approach: Mathematical and structural reasoning benchmarks · 1 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
make pipeline partitioning frozen-status aware (ablation)Cornstarch pipeline parallelism · MLLM-MMMiteration time (s): 173.01 -> 70.37Improved
make pipeline partitioning frozen-status aware (ablation)Cornstarch pipeline parallelism · MLLM-LLLiteration time (s): 247.79 -> 143.76Improved
make pipeline partitioning frozen-aware when encoder and LLM sizes are extremely different (ablation)Cornstarch pipeline parallelism · MLLM-LLS and MLLM-SSLiteration time (s): 66.20 -> 66.14No clear effect
add inter-GPU token balancing only to causal CP (ablation)context parallelism · Attention layer, 64k, LLM-Ltime (ms): 568.18 -> 610.67Hurt
add intra-GPU balancing only (ablation)context parallelism · Attention layer, 64k, LLM-Stime (ms): 243.44 -> 225.73Improved
combine inter-GPU and intra-GPU balancing (ablation)context parallelism · Attention layer, 64k, LLM-Stime (ms): 243.44 -> 204.95Improved
use Cornstarch all-gather CP instead of DCP ring CP at short sequence lengthcontext parallelism · Lambda attention, 32kattention layer time (ms): 46.36 -> 177.41Hurt
use Cornstarch all-gather CP instead of DCP ring CP at long sequence lengthcontext parallelism · Lambda attention, 256kattention layer time (ms): 1545.83 -> 1193.32Improved
use heuristic token assignment instead of DCP planningend-to-end training · Causal, 32kiteration time (s): 26.5 -> 5.7Improved
use Cornstarch instead of FSDP and Megatron*MLLM training · 24 A40 GPUs, various MLLM configsspeedup: 3.36× vs FSDP, 1.62× vs Megatron*Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy