Research map / Video / Video-language understanding
ReXTime: A Benchmark Suite for Reasoning-Across-Time in Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ReXTime, a video QA and moment-localization benchmark for reasoning across time (question and answer in different segments), built with an LLM-assisted pipeline plus a 9,695-sample machine-generated training set.
Problem
Existing video QA and moment retrieval benchmarks mostly test text-visual alignment and overlook cause-effect reasoning when the question and answer occur in different video segments.
Approach
Generate multiple-choice QA pairs from pairs of non-overlapping time-aligned events, categorized into sequential, cause-effect, and means-to-an-end relations via LLM-scored criteria, with LLM self-verification to cut human verification cost; models must also localize the answer span, and a QA-IoU measure quantifies across-time reasoning.
Key results
- GPT-4o reaches 73.67% VQA accuracy on the mini-test vs. 87.98% for humans, a 14.3-point gap.
- After LoRA fine-tuning on the 9,695 generated samples, VTimeLLM's VQA accuracy rises from 36.16% to 57.58% and TimeChat's from 40.04% to 49.46% (the paper's text reports different TimeChat zero-shot numbers).
- ReXTime has the lowest QA-mIoU (15.5%) and longest average certificate length (66.0 s) among compared datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune VTimeLLM on the ReXTime training dataset | VTimeLLM · ReXTime test | Accuracy: 36.16 -> 57.58 | Helped |
| Fine-tune TimeChat on the ReXTime training dataset | TimeChat · ReXTime test | Accuracy: 40.04 -> 49.46 | Helped |
| Fine-tune UniVTG on the ReXTime training dataset | UniVTG · ReXTime test | mIoU: 28.17 -> 34.63 | Helped |
| Fine-tune UniVTG on the ReXTime training dataset | UniVTG · ReXTime test | R@1 (IoU=0.3): 41.34 -> 53.48 | Helped |
| Fine-tune UniVTG on the ReXTime training dataset | UniVTG · ReXTime test | R@1 (IoU=0.5): 26.88 -> 34.53 | Helped |
| Use GPT-4o instead of human participants | ReXTime mini-test · ReXTime mini-test (300 samples) | Accuracy: 87.98 -> 73.67 | Hurt |
| Use GPT-4o instead of human participants | ReXTime mini-test · ReXTime mini-test (300 samples) | mIoU: 61.11 -> 36.28 | Hurt |
Surprising findings
- Frontier MLLMs localize the answer span poorly: GPT-4o gets 36.28 mIoU versus 61.11 for humans, and frontier MLLMs fall behind human performance on moment localization.
- Naively feeding video and captions to an MLLM yields logically incorrect QA pairs, and few-shot prompting improves correctness at the cost of diversity.
Limitations
- Only single continuous time spans for question and answer are covered.
- Noisy or inaccurate source captions and timestamps in ActivityNet and QVHighlights.
- Average ground-truth event durations are long, hindering fine-grained short-action analysis.
Experimental setup
Datasets: ActivityNet QVHighlights NExT-GQA Ego4D-NLQ
Benchmarks: ReXTime
Metrics: accuracy mIoU R@1 IoU=0.3 R@1 IoU=0.5 accuracy @ IoU>=0.5 QA-mIoU certificate length
Baselines: GPT-4o GPT-4V Gemini-1.5-Pro Claude3-Opus Reka-Core UniVTG CG-DETR VTimeLLM TimeChat LITA
Builds on: NExT-QA NExT-GQA Ego4D-NLQ QVHighlights ActivityNet
Closest papers by meaning
- RTV-Bench: Benchmarking MLLM Continuous Perception, Understanding and Reasoning through Real-Time Video (NeurIPS 2025 · similarity 0.88)
- TimE: A Multi-level Benchmark for Temporal Reasoning of LLMs in Real-World Scenarios (NeurIPS 2025 · similarity 0.88)
- Temporal Reasoning Transfer from Text to Video (ICLR 2025 · similarity 0.88)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.88)
- Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames (NeurIPS 2025 · similarity 0.87)
- MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.