Research map / Video / Video-language understanding
TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes three principles (multi-frame gain, frame order sensitivity, frame information disparity) to audit visual temporal reasoning benchmarks, and introduces TOMATO, a 1,484-question human-annotated video benchmark that exposes a large gap between MFMs and humans.
Problem
Existing video benchmarks for visual temporal reasoning in multimodal foundation models can often be solved from a single, a few, or out-of-order frames, so MFMs' temporal reasoning is likely overestimated.
Approach
Three quantitative principles/metrics (multi-frame gain, frame order sensitivity, frame information disparity) for auditing benchmarks, used to design TOMATO, a benchmark of human-annotated questions across six temporal tasks with self-recorded, simulated, and edited (counterfactual, reversed, mirrored) videos.
Key results
- Best open-source model Qwen2-VL-72B reaches 37.9% overall accuracy on TOMATO, versus 37.7% for GPT-4o and 95.2% for human video annotators, a 57.3% gap with the best model.
- TOMATO shows an average multi-frame gain of 66.3% versus under 5% on existing benchmarks, and frame order sensitivity of 34.1% versus under 8%.
- Average frame information disparity is 4.6% on TOMATO compared with over 27% on existing benchmarks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 16 ordered frames instead of one random frame | GPT-4o and Qwen2-VL on TOMATO · TOMATO (All) | Multi-Frame Gain (κ): 78.0; 84.0 | Helped |
| Use 16 ordered frames instead of one handpicked frame | GPT-4o and Qwen2-VL on TOMATO · TOMATO (200) | Multi-Frame Gain (κ): 72.1; 60.4 | Helped |
| Preserve chronological order instead of shuffling the 16 frames | GPT-4o and Qwen2-VL on TOMATO · TOMATO | Frame Order Sensitivity (τ): 46.2; 21.9 | Helped |
| Use random rather than handpicked single frames | GPT-4o and Qwen2-VL on TOMATO · TOMATO (200) | Frame Information Disparity (ρ): 4.9; 4.3 | Helped |
| Evaluate with human video input rather than MFM input | TOMATO models · TOMATO (All) | accuracy: 37.9 -> 95.2 | Helped |
| Use simulated human scenarios instead of real human scenarios | GPT-4o · TOMATO, human scenarios | accuracy change: 21.9% | Helped |
| Use simulated human scenarios instead of real human scenarios | Qwen2-VL-72B · TOMATO, human scenarios | accuracy change: 7.8% | Helped |
| Use counterfactual instead of non-counterfactual videos | GPT-4o · TOMATO counterfactual QAs | accuracy change: -37.7% | Hurt |
| Use counterfactual instead of non-counterfactual videos | Qwen2-VL-72B · TOMATO counterfactual QAs | accuracy change: -29.4% | Hurt |
| Provide zoomed-in instead of standard views | GPT-4o · TOMATO zoomed-in QAs | accuracy change: +6.0% | Helped |
| Provide zoomed-in instead of standard views | Qwen2-VL-7B · TOMATO zoomed-in QAs | accuracy change: +1.9% | Helped |
| Provide zoomed-in instead of standard views | Qwen2-VL-72B and Video-CCAM-v1.1 14B · TOMATO zoomed-in QAs | accuracy change: -4.2%; -1.0% | Hurt |
Surprising findings
- Performance of models plateaus beyond 8 frames on TOMATO even though human accuracy keeps improving with more frames.
- More capable models such as GPT-4o show a 37.7% drop on counterfactual (edited, reversed) videos, suggesting over-reliance on common sense.
Limitations
- Benchmark is small (1,484 questions) and results are evaluated mainly in zero-shot multiple-choice format.
- Some source videos come from existing datasets (Music-AVQA, CLEVRER, TGIF-QA, Perception Test) under non-commercial licenses.
- Scenario analysis subsets are small (e.g., 80 first-person QAs), so trends are described as preliminary.
Experimental setup
Datasets: Music-AVQA CLEVRER TGIF-QA Perception Test YouTube videos
Benchmarks: TOMATO VITATECS MVBench TempCompass ReXTime
Metrics: accuracy multi-frame gain frame order sensitivity frame information disparity
Baselines: GPT-4o Gemini 1.5 Pro Claude 3.5 Sonnet Qwen2-VL-72B LLaVA-Video-72B InternVL 2 40B Random Choice Frequent Choice
Builds on: VITATECS MVBench TempCompass ReXTime
Closest papers by meaning
- SciVideoBench: Benchmarking Scientific Video Reasoning in Large Multimodal Models (ICML 2026 · similarity 0.92)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.91)
- MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos (NeurIPS 2026 · similarity 0.90)
- MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (CVPR 2024 · similarity 0.90)
- Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognition (ICML 2024 · similarity 0.90)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.