Research map / Video / Video-language understanding
Time-R1: Post-Training Large Vision Language Model for Temporal Video Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Time-R1 post-trains Qwen2.5-VL with GRPO-style reinforcement learning using a tIoU and format reward, plus a curated 2.5K-sample difficulty-filtered dataset (TimeRFT) and a new 800-instance benchmark (TVGBench) for temporal video grounding.
Problem
Large vision-language models underperform on temporal video grounding when fine-tuned with SFT, which over-penalizes reasonable timestamp predictions and generalizes poorly; existing TVG benchmarks are also large and ill-suited for slow LVLM evaluation.
Approach
Apply RL with verifiable rewards (a timestamp-aware IoU plus a reasoning-template reward) to optimize LVLM grounding directly, combined with moderate-difficulty sample selection (Gaussian over IoU), multi-epoch dynamic hard sampling, and a brief LoRA cold start with CoT captions.
Key results
- Time-R1 (7B, zero-shot) reaches R1@0.7 of 35.3 on Charades-STA and 21.4 on ActivityNet, and 16.4 R1@0.7 on TVGBench, beating prior 7B LVLMs such as VideoChat-Flash (27.6) and TRACE (14.6) on those metrics.
- Time-R1-7B with RL improves ActivityNet mIoU from 16.3 to 29.2 while full-parameter SFT degrades it to 15.4 and SFT-LoRA reaches 25.9.
- On VideoMME long split, RL raises accuracy from 53.0 to 54.2 whereas SFT-LoRA drops it to 51.7.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Post-train Qwen2.5-VL-7B with Time-R1 RL | Qwen2.5-VL-7B · TVGBench | R1@0.7: 12.8 -> 16.4 | Helped |
| Add Multi-Epoch training (ablation) | Time-R1 · TVGBench | R1@0.7: 13.2 -> 14.2 | Helped |
| Add Gaussian Filtering to the no-component setup (ablation) | Time-R1 · TVGBench | R1@0.7: 13.2 -> 13.4 | Helped |
| Add tIoU reward to the no-component setup (ablation) | Time-R1 · TVGBench | R1@0.7: 13.2 -> 12.9 | Hurt |
| Combine tIoU reward with Multi-Epoch training (ablation) | Time-R1 · TVGBench | R1@0.7: 13.2 -> 16.4 | Helped |
| Add Sample Filtering to the tIoU + Gaussian Filtering + Multi-Epoch setup (ablation) | Time-R1 · TVGBench | R1@0.5: 28.5 -> 29.4 | Helped |
| Use Gaussian filtering (0.3) rather than random filtering (ablation) | Time-R1 · TVGBench | R1@0.3: 39.4 -> 41.6 | Helped |
| Use Gaussian filtering (0.5) rather than random filtering (ablation) | Time-R1 · TVGBench | R1@0.7: 16.4 -> 16.0 | Hurt |
| Use Gaussian filtering (0.7) rather than random filtering (ablation) | Time-R1 · TVGBench | R1@0.7: 16.4 -> 15.5 | Hurt |
| Use uniform rather than random filtering (ablation) | Time-R1 · TVGBench | R1@0.7: 16.4 -> 15.9 | Hurt |
| Add CoT to the no-KL GRPO setup (ablation) | Time-R1 · TVGBench | R1@0.5: 29.1 -> 29.5 | Helped |
| Add KL regularization to the no-CoT GRPO setup (ablation) | Time-R1 · TVGBench | R1@0.5: 29.1 -> 27.4 | Hurt |
Surprising findings
- Full-parameter SFT on only 2.5K samples degrades generalization across tasks, while RL preserves or improves it.
- Without KL regularization, the model in the No-KL+CoT setting often skips the thinking process and jumps directly to answers.
Limitations
- Slow training and inference due to large model size and autoregressive generation.
- Low frame sampling rate (2 FPS, low resolution) may lose fine-grained motion information.
- Cannot handle ultra-long videos such as full-length movies.
Experimental setup
Datasets: Charades-STA ActivityNet YT-Temporal DiDeMo QuerYD InternVid HowTo100M VTG-IT TimeIT TimePro HTStep LongVid
Benchmarks: Charades-STA ActivityNet TVGBench MVBench TempCompass EgoSchema VideoMME
Metrics: R1@0.3 R1@0.5 R1@0.7 mIoU accuracy
Baselines: TimeChat TimeSuite TRACE VideoChat-Flash Gemini-2.5-Pro HawkEye VTimeLLM SFT SFT-LoRA
Builds on: GRPO DAPO DeepSeek-R1 Qwen2.5-VL
Closest papers by meaning
- Uncertainty-quantified Rollout Policy Adaptation for Unlabelled Cross-domain Video Temporal Grounding (NeurIPS 2025 · similarity 0.90)
- TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning (ICLR 2025 · similarity 0.90)
- ReVisionLLM: Recursive Vision-Language Model for Temporal Grounding in Hour-Long Videos (CVPR 2025 · similarity 0.90)
- Universal Video Temporal Grounding with Generative Multi-modal Large Language Models (NeurIPS 2025 · similarity 0.89)
- Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Grounding (NeurIPS 2025 · similarity 0.89)
- Video-R1: Reinforcing Video Reasoning in MLLMs (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.