Research map / Video / Video-language understanding
Unhackable Temporal Reward for Scalable Video MLLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Frames temporal hacking as reward hacking in video MLLM training, proposes the Temporal Perplexity (TPL) score as a diagnostic and the UTR data/task construction method (spatiotemporal attribute trajectories with bidirectional querying), yielding Video-UTR (Qwen2-7B based).
Problem
Video MLLMs show an "anti-scaling law" where more data and larger models hurt temporal video understanding, because models shortcut by attending to a few frames (temporal hacking).
Approach
Recasts video-language pretraining as an RL reward-hacking problem, defines a TPL score (perplexity gap between full-video and single-keyframe context) to measure hackability, and mitigates it with UTR: spatiotemporal attribute trajectories from expert detectors/trackers used in bidirectional query-answer SFT data that forces reading all frames.
Key results
- Video-UTR (7B) reaches 58.78% on MVBench and 59.67% on TempCompass, ranking first on 4 of 7 video benchmarks while using about 1.1M samples.
- On image benchmarks Video-UTR scores 76.6 on MMBench vs 72.1 for LLaVA-NeXT-Img and 88.9 on POPE.
- Ablation: removing UTR-Data drops MVBench from 58.78 to 54.63 and TGIF-QA from 56.44 to 54.74.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add UTR-Data (ablation) | Video-UTR · MVBench | accuracy: 54.63 -> 58.78 | Helped |
| add UTR-Data (ablation) | Video-UTR · TGIF-QA | accuracy: 54.74 -> 56.44 | Helped |
| add UTR-Data (ablation) | Video-UTR · ActivityNet-QA | accuracy: 54.15 -> 55.00 | Helped |
| add bidirectional querying (ablation) | Video-UTR · MVBench | accuracy: 58.45 -> 58.78 | Helped |
| add bidirectional querying (ablation) | Video-UTR · TGIF-QA | accuracy: 56.11 -> 56.44 | Helped |
| add bidirectional querying (ablation) | Video-UTR · ActivityNet-QA | accuracy: 54.21 -> 55.00 | Helped |
| replace UTR-Data with more VideoChat2 data (ablation) | Video-UTR · TGIF-QA | accuracy: 56.44 -> 53.39 | Hurt |
| replace UTR-Data with more VideoChat2 data (ablation) | Video-UTR · MVBench | accuracy: 58.78 -> 57.65 | Hurt |
| increase UTR-Data size from 0K to 325K (ablation) | Video-UTR · MVBench | accuracy: 54.63 -> 58.78 | Helped |
| increase UTR-Data size from 0K to 325K (ablation) | Video-UTR · TempCompass | score: 58.88 -> 59.67 | Mixed |
| increase frame length from 8 to 32 (ablation) | Video-UTR · MVBench | accuracy: 50.93 -> 51.40 | Mixed |
Surprising findings
- Adding an equal amount of extra VideoChat2 data did not improve results, suggesting data construction matters more than volume.
Limitations
- UTR depends on the accuracy of expert detectors and trackers for attribute extraction.
- Gains are limited on knowledge-heavy or single-frame questions in VideoMME.
- Evaluation relies partly on GPT-3.5 judged scores and reproduced numbers for some baselines.
Experimental setup
Datasets: HowTo100M MeViS LaMOT VideoChat2 data LLaVA-NeXT-SFT BLIP-558K WebVid
Benchmarks: MVBench TempCompass VideoMME MSVD-QA MSRVTT-QA TGIF-QA ActivityNet-QA MM-Vet MMBench MMMU MME LLaVA-Wild
Metrics: accuracy GPT score
Baselines: VideoChat2 VideoChat Video-ChatGPT Video-LLaVA VideoLLaMA2 PLLaVA LLaVA-NeXT-Video LLaVA-OneVision
Builds on: LLaVA-NeXT-Video VideoChat2 Merlin ByteTrack
Closest papers by meaning
- VideoSSR: Video Self-Supervised Reinforcement Learning (CVPR 2026 · similarity 0.86)
- Before It Fades: Reinforcing Temporal Representations at Inference Time in VideoLLMs (NeurIPS 2026 · similarity 0.85)
- TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning (ICLR 2025 · similarity 0.84)
- On the Consistency of Video Large Language Models in Temporal Comprehension (CVPR 2025 · similarity 0.84)
- VideoRFT: Incentivizing Video Reasoning Capability in MLLMs via Reinforced Fine-Tuning (NeurIPS 2025 · similarity 0.83)
- Video-R1: Reinforcing Video Reasoning in MLLMs (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.