Research map / Large language models / Reasoning and chain of thought

Heterogeneous Parallelism for Multimodal Large Language Model Training

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes heterogeneous parallelism in Megatron-LM letting each module in a multimodal LLM graph use its own TP/CP/PP/DP/EP layout, via boundary communicators with colocated and non-colocated modes.

Problem

A single LLM-centric TP/DP/PP/CP layout forces modality encoders to inherit LLM sharding and placement choices, bottlenecking multimodal training throughput.

Approach

Autograd-aware boundary communicators convert activations between independent per-module layouts in forward and invert the transform for gradients in backward, supporting shared-GPU (colocated) and disjoint-rank (non-colocated) placements, with a graph-aware 1F1B schedule and a three-phase colocated schedule.

Key results

• Colocated heterogeneous execution improves TFLOPS/GPU by up to 49.3% over homogeneous at 64K context (1B+7B).
• Non-colocated execution improves aggregate tokens/s by up to 13.0% and TFLOPS/GPU by up to 9.6% for large encoders with a 120B LLM.
• FP32 step-level parity and LLaVA-style loss curves overlap across homogeneous, colocated, and non-colocated layouts.

Surprising findings

• Colocated heterogeneity converges back to homogeneous for the largest encoders (e.g., 12B+120B shows -8.0% TFLOPS/GPU), where non-colocation is preferred.

Limitations

• Evaluated mainly on throughput sweeps with LLaVA-style workloads; convergence validated only on a small LLaVA-style setup.
• Non-colocated gains are conservative in TFLOPS/GPU since they average over an extra encoder island; gains vary by regime.
• Colocated gains diminish as memory pressure grows and the best mode depends on workload.

Experimental setup

Datasets: LLaVA-Pretrain

Benchmarks: LLaVA-Pretrain loss curves throughput sweeps (TFLOPS/GPU, tokens/s)

Metrics: TFLOPS/GPU tokens/s step time peak memory loss parity FP32 step-level parity

Baselines: homogeneous shared-layout training (H)

Builds on: Megatron-LM 1F1B pipeline scheduling LLaVA

Closest papers by meaning

• EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism (ICML 2024 · similarity 0.88)
• ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelism (NeurIPS 2025 · similarity 0.88)
• OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging (ICLR 2026 · similarity 0.88)
• Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · similarity 0.88)
• Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.88)
• DuoGPT: Training-free Dual Sparsity through Activation-aware Pruning in LLMs (NeurIPS 2025 · similarity 0.88)