Research map / Vision-language and multimodal / Multimodal LLMs
OmniBal: Towards Fast Instruction-Tuning for Vision-Language Models via Omniverse Computation Balance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OmniBal, a three-part (data, model, memory) computation-balancing framework for 3D-parallel instruction tuning of vision-language models, reporting about 1.8-2x training speedup over InternVL-Chat baselines.
Problem
Large-scale 3D-parallel instruction tuning of VLMs suffers computation imbalance across devices and pipeline stages because the ViT and LLM parts are heterogeneous and inputs vary in image count and text length, causing idle bubbles and over-aggressive recomputation.
Approach
Jointly rebalances computation across data (ISF grouping into balanced dynamic mini-batches within and across devices), model (search-based pipeline partition scored by stage-time variance and communication volume), and memory (per-stage adaptive recomputation that uses the freed memory).
Key results
- On InternVL-Chat-1.5 6+20B with Megatron-DeepSpeed, GPU days drop from 61.8 to 21.3 (1.83x speed-up), with benchmark scores comparable to the baseline.
- On the 6+34B model with Megatron-DeepSpeed, GPU days drop from 75.4 to 30.5 (1.80x), and with DeepSpeed ZeRO-3 on 6+20B from 38.9 to 25.3 (1.54x).
- Across model combinations (e.g., EVA-CLIP-1B to 18B with InternLM2-20B), speed-ups range from 1.9x to 2.2x, and up to 3.2x with Qwen1.5-110B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| balance data, model partitioning, and memory | InternVL-Chat-1.5 6+20B with DeepSpeed · InternVL-Chat-1.5 6+20B | GPU Days: 38.9 -> 25.3 | Helped |
| balance data, model partitioning, and memory | InternVL-Chat-1.5 6+20B with Megatron · InternVL-Chat-1.5 6+20B | GPU Days: 61.8 -> 21.3 | Helped |
| balance data, model partitioning, and memory | InternVL-Chat-1.5-Plus 6+34B with DeepSpeed · InternVL-Chat-1.5-Plus 6+34B | GPU Days: 54.3 -> 35.5 | Helped |
| balance data, model partitioning, and memory | InternVL-Chat-1.5-Plus 6+34B with Megatron · InternVL-Chat-1.5-Plus 6+34B | GPU Days: 75.4 -> 30.5 | Helped |
| balance data, model partitioning, and memory | InternVL-Chat-1.5 6+20B with Megatron · InternVL-Chat-1.5 6+20B, MMB-EN/CN | MMB-EN/CN: 79.5/77.7 -> 78.6/77.5 | Hurt |
| balance data, model partitioning, and memory | InternVL-Chat-1.5 6+20B with Megatron · InternVL-Chat-1.5 6+20B, ChartQA | ChartQA: 87.3 -> 86.7 | Hurt |
| balance data, model partitioning, and memory | InternVL-Chat-1.5 6+20B with Megatron · InternVL-Chat-1.5 6+20B, AI2D | AI2D: 71.6 -> 70.9 | Hurt |
| balance data, model partitioning, and memory | InternVL-Chat-1.5 6+20B with Megatron · InternVL-Chat-1.5 6+20B, MMVet | MMVet: 45.0 -> 48.5 | Helped |
| add data balance (ablation) | InternVL-Chat-1.5 6+20B with Megatron · InternVL-Chat-1.5 6+20B | GPU Days: 61.8 -> 51.9 | Helped |
| add model balance to data balance (ablation) | InternVL-Chat-1.5 6+20B with Megatron · InternVL-Chat-1.5 6+20B | GPU Days: 51.9 -> 29.0 | Helped |
| add memory balance to data and model balance (ablation) | InternVL-Chat-1.5 6+20B with Megatron · InternVL-Chat-1.5 6+20B | GPU Days: 29.0 -> 21.3 | Helped |
| use ISF balanced dynamic mini-batches instead of random grouping (ablation) | InternVL-Chat-1.5 6+20B · Data balance comparison | GPU Days: 61.8 -> 51.9 | Helped |
Surprising findings
- Switching from DeepSpeed ZeRO-3 to Megatron-DeepSpeed 3D parallelism slows training without balancing (38.9 to 61.8 GPU days), so 3D parallelism only pays off after balancing.
- Padding-based baselines with length or device grouping alone give limited gains compared with the combined ISF grouping, which reaches pad ratio 0.
Limitations
- Q_v and Q_t thresholds must be set from dataset statistics and are dataset-dependent.
- Balanced model partitioning relies on candidate search and profiling of layer times, which adds overhead and needs re-tuning per hardware and model.
- Main experiments use the InternVL-Chat-1.2 dataset as a substitute because the InternVL-Chat-1.5 data was unavailable; speed-up is measured in GPU days only, with limited accuracy evaluation.
Experimental setup
Datasets: InternVL-Chat-1.2 (1.2M) LLaVA-665K LCS-558K
Benchmarks: MMBench-EN MMBench-CN ChartQA AI2D MMVet MME
Metrics: GPU days speed-up ratio Pad Ratio Dist Ratio benchmark accuracy
Baselines: DeepSpeed ZeRO-3 training of InternVL-Chat-1.5 Megatron-DeepSpeed 3D parallel training parameter-based partitioning layer-based partitioning profile-based partitioning (DreamPipe) length-group batching device-group batching
Builds on: Megatron-DeepSpeed DeepSpeed ZeRO GPipe PipeDream AdaPipe
Closest papers by meaning
- VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Models (CVPR 2025 · similarity 0.88)
- Connecting the Dots: Collaborative Fine-tuning for Black-Box Vision-Language Models (ICML 2024 · similarity 0.87)
- Hawaii: Hierarchical Visual Knowledge Transfer for Efficient Vision-Language Models (NeurIPS 2025 · similarity 0.87)
- Towards Self-Refinement of Vision-Language Models with Triangular Consistency (NeurIPS 2025 · similarity 0.86)
- VIGC: Visual Instruction Generation and Correction (AAAI 2024 · similarity 0.86)
- Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.