Research map / Video / Video-language understanding
Scaling RL to Long Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds LongVideo-Reason (104K long-video reasoning QA pairs), trains LongVILA-R1-7B via CoT-SFT then GRPO-based RL, and introduces MR-SP, a sequence-parallel RL training system with cached video embeddings and vLLM rollouts.
Problem
Enabling reasoning in vision-language models over long videos is hampered by scarce long-video reasoning data and the high memory and compute cost of RL on hundreds to thousands of frames.
Approach
Combine a newly constructed long-video CoT dataset with a two-stage CoT-SFT plus GRPO pipeline, and a multi-modal sequence-parallel RL system (MR-SP) that caches gathered video embeddings for reuse across rollouts and shards prefilling across GPUs.
Key results
- LongVILA-R1-7B reaches 65.1% (w/o subtitles) and 71.1% (w/ subtitles) on VideoMME, outperforming LongVILA-7B (60.1%/65.1%).
- LongVILA-R1-7B averages 72.0% on LongVideo-Reason-eval, above Video-R1-7B (68.1%) and Gemini-1.5-Pro (69.3%).
- MR-SP achieves up to 2.1x speedup over plain RL at 512 frames on 8xA100 and supports 3,600-frame RL on a single node.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply LongVILA-R1 training (CoT-SFT and RL) instead of the LongVILA baseline | LongVILA-7B · ActivityNet-QA test | accuracy: 59.5 -> 64.8 | Helped |
| Apply LongVILA-R1 training (CoT-SFT and RL) instead of the LongVILA baseline | LongVILA-7B · LongVideoBench val | accuracy: 57.1 -> 58.0 | Helped |
| Apply LongVILA-R1 training (CoT-SFT and RL) instead of the LongVILA baseline | LongVILA-7B · PerceptionTest val | accuracy: 58.1 -> 68.9 | Helped |
| Apply LongVILA-R1 training (CoT-SFT and RL) instead of the LongVILA baseline | LongVILA-7B · NExT-QA mc | accuracy: 80.7 -> 81.5 | Helped |
| Apply LongVILA-R1 training (CoT-SFT and RL) instead of the LongVILA baseline | LongVILA-7B · VNBench val | accuracy: 63.0 -> 75.5 | Helped |
| Apply LongVILA-R1 training (CoT-SFT and RL) instead of the LongVILA baseline | LongVILA-7B · VideoMME w/o subtitles | accuracy: 60.1 -> 65.1 | Helped |
| Apply LongVILA-R1 training (CoT-SFT and RL) instead of the LongVILA baseline | LongVILA-7B · VideoMME with subtitles | accuracy: 65.1 -> 71.1 | Helped |
| Apply LongVILA-R1 training instead of the LongVILA baseline | LongVILA-7B · LongVideo-Reason-eval overall | accuracy: 62.7 -> 72.0 | Helped |
| Add reasoning training (ablation) | LongVILA-1.5B · LongVideo-Reason-eval, 16 frames | accuracy: 55.7 -> 55.9 | Helped |
| Add reasoning training (ablation) | LongVILA-1.5B · LongVideo-Reason-eval, 32 frames | accuracy: 56.4 -> 56.7 | Helped |
| Add reasoning training (ablation) | LongVILA-1.5B · LongVideo-Reason-eval, 64 frames | accuracy: 58.1 -> 61.9 | Helped |
| Add reasoning training (ablation) | LongVILA-1.5B · LongVideo-Reason-eval, 128 frames | accuracy: 60.5 -> 62.6 | Helped |
Surprising findings
- Skipping CoT-SFT and applying RL directly lowers accuracy (52.4 vs 58.1 baseline in the ablation).
- LongVILA-1.5B without reasoning degrades at 512 frames (60.2 vs 60.7 at 256), while the reasoning variant keeps improving.
Limitations
- Scaling to much longer sequences, audio, or larger batch sizes requires multi-GPU distributed training, which limits accessibility on limited resources.
- Data generation consumed about 80,000 H100 GPU hours, and the LongVideo-Reason-eval benchmark is small (1K samples).
Experimental setup
Datasets: LongVideo-Reason Shot2Story Video-R1 datasets open-source video QA datasets (102K)
Benchmarks: ActivityNet-QA LongVideoBench PerceptionTest NExT-QA VNBench VideoMME LongVideo-Reason-eval
Metrics: accuracy
Baselines: LongVILA-7B Video-R1-7B Gemini-1.5-Pro GPT-4o LLaVA-OV-7B Qwen2.5-VL-7B
Builds on: GRPO LongVILA DeepSeek-R1 Video-R1 vLLM
Closest papers by meaning
- Video-R1: Reinforcing Video Reasoning in MLLMs (NeurIPS 2025 · similarity 0.91)
- Incentivizing Versatile Video Reasoning in MLLMs via Data-Efficient Reinforcement Learning (CVPR 2026 · similarity 0.91)
- LongVILA: Scaling Long-Context Visual Language Models for Long Videos (ICLR 2025 · similarity 0.90)
- VideoReasonBench: Can MLLMs Perform Vision-Centric Complex Video Reasoning? (ICLR 2026 · similarity 0.89)
- Unleashing Hour-Scale Video Training for Long Video-Language Understanding (NeurIPS 2025 · similarity 0.89)
- Time-R1: Post-Training Large Vision Language Model for Temporal Video Grounding (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.