Research map / Large language models / Instruction and fine-tuning
DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPO
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DeepVideo-R1, a VideoLLM trained with Reg-GRPO (regressing group-normalized advantages instead of clipped policy-gradient loss) plus difficulty-aware data augmentation, evaluated on video reasoning benchmarks.
Problem
GRPO for VideoLLMs relies on clipping/min safeguards that suppress gradients and suffers from vanishing advantages when samples are too easy or too hard for the current model.
Approach
Reformulate the GRPO objective as a regression: the policy's log-ratio-based normalized advantage is regressed onto the reward-based group-normalized advantage (no clipping/min), and augment inputs adaptively (noise for easy samples, reasoning hints for hard samples) to keep advantage variance nonzero.
Key results
- On SEED-Bench-R1, DeepVideo-R1-3B improves over Qwen2.5-VL-3B+GRPO overall by +8.5, +10.1, and +8.6 points on L1, L2, L3 (e.g., L1 48.1 vs 39.6).
- On NExTGQA, DeepVideo-R1-7B reaches 36.8 mIoU and 72.5 Acc@QA, beating VideoChat-R1 by +4.4 mIoU and +2.3 Acc@QA.
- On LongVideoBench, DeepVideo-R1-3B reaches 51.1 overall vs 43.2 for Qwen2.5-VL-3B+GRPO; Reg-GRPO alone improves L3 from 35.4 (GRPO) to 39.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with DeepVideo-R1 instead of GRPO | Qwen2.5-VL-3B · SEED-Bench-R1 L1 (Daily life) | score: 39.6 -> 48.1 | Helped |
| train with DeepVideo-R1 instead of GRPO | Qwen2.5-VL-3B · SEED-Bench-R1 L2 (Daily life) | score: 41.0 -> 51.1 | Helped |
| train with DeepVideo-R1 instead of GRPO | Qwen2.5-VL-3B · SEED-Bench-R1 L3 Overall | score: 35.4 -> 44.0 | Helped |
| train with DeepVideo-R1 instead of GRPO | Qwen2.5-VL-3B · LongVideoBench Overall | score: 43.2 -> 48.4 | Helped |
| train with DeepVideo-R1 instead of the base model | Qwen2.5-VL-3B · Video-MME (wo sub) | score: 54.4 -> 51.1 | Hurt |
| add difficulty-aware augmentation to GRPO (ablation) | Qwen2.5-VL-3B + GRPO · SEED-Bench-R1 L1 | score: 39.6 -> 41.7 (+2.1) | Helped |
| replace GRPO with Reg-GRPO without difficulty-aware augmentation (ablation) | Qwen2.5-VL-3B · SEED-Bench-R1 L3 | score: 35.4 -> 39.5 (+4.1) | Helped |
| use relative rather than absolute difficulty measurement (ablation) | DeepVideo-R1 data augmentation · SEED-Bench-R1 L3 | score: 40.7 -> 44.0 | Helped |
| use both difficulty-increasing and difficulty-decreasing augmentation (ablation) | DeepVideo-R1 · SEED-Bench-R1 L3 | score: 36.3 -> 44.0 (+7.7) | Helped |
| use adaptive rather than fixed augmentation scaling (ablation) | DeepVideo-R1 · SEED-Bench-R1 L2 | score: 48.6 -> 51.1 | Helped |
Surprising findings
- GRPO was run without safeguards (clipping/min) because the authors found them unhelpful empirically, which is part of their motivation.
- Reward regression (direct reward regression) performs worse than the untuned base on some splits (L1 32.5 vs 31.3 base is only slightly higher, but well below GRPO).
Limitations
- High computational and memory requirements of VideoLLMs.
- Possible overlap between pretraining data and evaluation benchmarks.
- Single runs without error bars, so significance of gains is unknown.
Experimental setup
Datasets: SEED-Bench-R1 Video-R1 training set NExTGQA
Benchmarks: SEED-Bench-R1 LongVideoBench VSI-Bench Video-MMMU MMVU MVBench TempCompass Video-MME NExTGQA
Metrics: accuracy mIoU Acc@QA
Baselines: SFT GRPO DPO Online DPO REINFORCE RLOO REBEL Reward-Regression VideoChat-R1 VideoLLaMA3-7B InternVL3-14B
Builds on: GRPO PPO DPO REBEL Video-R1
Closest papers by meaning
- Video-R1: Reinforcing Video Reasoning in MLLMs (NeurIPS 2025 · similarity 0.91)
- GVPO: Group Variance Policy Optimization for Large Language Model Post-Training (NeurIPS 2025 · similarity 0.89)
- TempSamp-R1: Effective Temporal Sampling with Reinforcement Fine-Tuning for Video LLMs (NeurIPS 2025 · similarity 0.89)
- Dr. Seg: Revisiting GRPO Training for Visual Large Language Models through Perception-Oriented Design (CVPR 2026 · similarity 0.88)
- Fast-Slow Thinking GRPO for Large Vision-Language Model Reasoning (NeurIPS 2025 · similarity 0.88)
- All Roads Lead to Rome: Incentivizing Divergent Thinking in Vision-Language Models (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.