Research map / Video / Video-language understanding
Impossible Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces IPV-Bench, a benchmark of impossible/counterfactual videos and prompts that evaluates text-to-video generation (prompt following and visual quality via IPV-Score) and Video-LLM understanding (judgment, MCQA, open-ended QA).
Problem
Synthetic video data and video models are largely evaluated on real-world-plausible content; it is unclear whether video generators can follow prompts describing impossible events and whether Video-LLMs can understand such videos, which require temporal reasoning and world knowledge.
Approach
A four-domain taxonomy of impossible scenes (physical, biological, geographical, social laws) drives a 260-prompt generation suite and a 902-video benchmark with human annotations; generation is scored by a joint visual-quality × impossible-prompt-following metric (IPV-Score), and understanding is tested via judgment, multiple-choice, and LLM-judged open-ended QA.
Key results
- Best text-to-video model Mochi 1 achieves an IPV-Score of 37.3%, with others lower (e.g., Hailuo 36.2%, Luma 14.3%).
- On MCQA, LLaVA-Next reaches 86.4% overall accuracy, above GPT-4o (79.7%) and Gemini-1.5-Flash (84.4%); Video-LLaVA gets 26.8%.
- On open-ended QA, GPT-4o scores highest (49.1 with GPT-4o evaluator, 45.1 with Claude evaluator), and temporal-focused videos score lower than spatial ones for all models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the three-step prompt strategy for GPT-4o impossible-prompt evaluation | GPT-4o evaluation of Kling-generated videos · Kling-generated videos | human alignment: 0.72 -> 0.80; 11.1% | Helped |
| Evaluate impossible video understanding on temporal-focused rather than spatial-focused videos | Video-LLMs · IPV-Vid spatial versus temporal videos | MCQA and OpenQA scores | Hurt |
| Use GPT-4o rather than Claude-3.5-Sonnet as the OpenQA evaluator for GPT-4o | GPT-4o impossible-video OpenQA evaluation · IPV-Vid overall OpenQA | semantic alignment: 45.1 -> 49.1 | Helped |
| Use GPT-4o rather than Claude-3.5-Sonnet as evaluator for most other models | Video-LLMs' OpenQA evaluation · IPV-Vid overall OpenQA | semantic alignment | Hurt |
Surprising findings
- Open-source Mochi 1 has higher impossible-prompt-following than many proprietary models, while Luma has high visual quality but low prompt following.
- Top video-understanding models (LLaVA-Next, GPT-4o) are image-based, suggesting temporal modules matter more than context length or frame rate.
Limitations
- Human annotation and GPT-4o-based distractor generation and evaluation may introduce hallucination or bias; GPT-4o self-evaluation bias noted.
- Judgment task results are biased (models often answer mostly yes or no), and GPT-4o refused to answer yes/no for it.
Experimental setup
Datasets: OpenVid
Benchmarks: IPV-Bench VBench
Metrics: IPV-Score visual quality prompt following accuracy F1 multiple-choice accuracy open-ended alignment score
Baselines: VBench VideoPhy PhyGenBench LOKI GPT-4o Gemini-1.5-Flash LLaVA-Next Qwen2-VL Video-LLaVA Oryx InternVL-2.5 NVILA
Builds on: VBench
Closest papers by meaning
- WorldReasonBench: Human-Aligned Stress Testing of Video Generators as Future World-State Predictors (NeurIPS 2026 · similarity 0.88)
- Video-Bench: Human-Aligned Video Generation Benchmark (CVPR 2025 · similarity 0.87)
- VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understanding (NeurIPS 2025 · similarity 0.87)
- Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation (ICML 2025 · similarity 0.86)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.86)
- VideoPhy: Evaluating Physical Commonsense for Video Generation (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.