Research map / Vision-language and multimodal / Multimodal LLMs
UNIVERSAL AND EFFICIENT LOADING BALANCING FOR RL TRAINING OF LARGE MULTIMODAL MODELS
Zerui Wang, Qinghao Hu, Chang Chen, Jiecheng Zhou, Haojie Duanmu, Xingcheng Zhang, Peng Sun, Dahua Lin
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Addresses system-level inefficiencies in RL training of vision-language models caused by heterogeneous multimodal data, proposing a load-balancing approach for the RL pipeline.
Problem
RL alignment of large VLMs is hampered by system bottlenecks (centralized data loading I/O, inference rollouts, model updates) due to extreme heterogeneity of multimodal data.
Approach
Load balancing across the RL training pipeline to handle heterogeneous multimodal data efficiently; specific mechanism not stated in the available abstract.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add ShadowLoader (distributed metadata-driven data loading) (ablation) | veRL · Image-Heavy, MiMo-VL-7B-RL, 128 H800 GPUs | data loading time: 375.0 s -> 3.9 s | Improved |
| add ShadowLoader (ablation) | veRL · Image-Heavy, MiMo-VL-7B-RL, 128 H800 GPUs | step time: 656.2 s -> 122.6 s | Improved |
| add FlexUlysses alone without ShadowLoader (ablation) | veRL · Image-Heavy, MiMo-VL-7B-RL, 128 H800 GPUs | end-to-end throughput: 1.00x -> 1.17x | Improved |
| add FlexUlysses on top of ShadowLoader (ablation) | veRL+ShadowLoader · Image-Heavy, MiMo-VL-7B-RL, 128 H800 GPUs | step time: 122.6 s -> 85.7 s | Improved |
| use FlexUlysses adaptive sharding instead of fixed-degree Ulysses-SP (SP=8) | FlexRL · Image-Heavy, 7B, 32 GPUs | end-to-end throughput speedup vs veRL: 1.13x -> 1.50x | Improved |
| increase fixed Ulysses SP degree (ablation) | Ulysses-SP · Image-Heavy, 7B, 32 GPUs | attention balance ratio: 6.70 -> 1.12 (SP=4) | Improved |
| FlexUlysses sharding for rollout and training when data loading is not the bottleneck (ablation) | veRL+ShadowLoader · Image-Heavy, MiMo-VL-7B-RL, 128 H800 GPUs | update actor time: 24.3 s -> 9.5 s | Improved |
| switch from Ulysses-SP to FlexRL on larger models and video-heavy data | FlexRL · Only-Video, 7B, 128 GPUs | throughput: 50.67 -> 405.44 | Improved |
| generation (rollout) stage with FlexUlysses added to ShadowLoader (ablation) | veRL+ShadowLoader · Image-Heavy, MiMo-VL-7B-RL, 128 H800 GPUs | generation time: 64.1 s -> 64.4 s | No clear effect |
Closest papers by meaning
- Scaling RL to Long Videos (NeurIPS 2025 · similarity 0.90)
- D-VLA: A High-Concurrency Distributed Asynchronous RL Framework for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.90)
- PeRL: Permutation-Enhanced Reinforcement Learning for Interleaved Vision-Language Reasoning (NeurIPS 2025 · similarity 0.88)
- What Can RL Bring to VLA Generalization? An Empirical Study (NeurIPS 2025 · similarity 0.88)
- RL makes MLLMs see better than SFT (ICLR 2026 · similarity 0.88)
- EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Models (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
