Research map / Vision-language and multimodal / Cross-modal learning
OmniScale: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoo
Qianli Ma, Yaowei Zheng, Zhelun Shi, Zhongkai Zhao, Bin Jia, Ziyue Huang, Zhiqi Lin, Youjie Li, Jiacheng Yang, Yanghua Peng, Zhi Zhang, Xin Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use VeOmni instead of TorchTitan | FSDP training · Qwen2-7B FSDP 8k, 128 GPUs | throughput: 6,639 -> 6,940 | Improved |
| use VeOmni instead of TorchTitan | FSDP+SP4 · Qwen2-7B 32k, 128 GPUs | throughput: 3,727 -> 4,629 | Improved |
| use VeOmni instead of TorchTitan | FSDP+SP8 · Qwen2.5-32B 32k, 128 GPUs | throughput: 821 -> 1,115 | Improved |
| use VeOmni instead of TorchTitan | FSDP+SP4 · Qwen2-72B 32k, 128 GPUs | throughput: 497 -> 592 | Improved |
| use VeOmni instead of TorchTitan | FSDP · Qwen2-7B 32k, 128 GPUs | MFU (%): 39.68 -> 44.42 | Improved |
| use VeOmni instead of TorchTitan | FSDP · Qwen2-7B 32k, 128 GPUs | memory (GB): 71.81 -> 31.41 | Improved |
| use VeOmni instead of TorchTitan | FSDP · Qwen2-7B 64k, 128 GPUs | trainability: OOM -> 3,725 | Improved |
| use VeOmni instead of TorchTitan | FSDP+SP4 · Qwen2-72B 8k, 128 GPUs | memory (GB): 54.08 -> 58.62 | Hurt |
| use VeOmni instead of TorchTitan | FSDP+SP8 · Qwen2-72B 128k, 128 GPUs | trainability: OOM -> 303 | Improved |
| increase sequence parallel size from 4 to 8 (ablation) | VeOmni · Qwen2-72B 128k, 128 GPUs | trainability: OOM -> 303 | Improved |
| increase sequence length from 8k to 128k with SP4 (ablation) | VeOmni · Qwen3-30B-A3B EP8, 128 GPUs | MFU (%): 3.02 -> 17.92 | Improved |
| add SP4 on top of FSDP+EP8 at 8k (ablation) | VeOmni · Qwen3-30B-A3B 8k, 128 GPUs | throughput: 2,776 -> 1,216 | Hurt |
Closest papers by meaning
- OmniBench: Towards The Future of Universal Omni-Language Models (NeurIPS 2025 · similarity 0.89)
- HaploVL: A Single-Transformer Baseline for Multi-Modal Understanding (ICML 2025 · similarity 0.87)
- OmnixR: Evaluating Omni-modality Language Models on Reasoning across Modalities (ICLR 2025 · similarity 0.86)
- Heterogeneous Parallelism for Multimodal Large Language Model Training (NeurIPS 2026 · similarity 0.86)
- OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models (ICLR 2024 · similarity 0.86)
- LLaMA-Omni: Seamless Speech Interaction with Large Language Models (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
