Research map / Large language models / Retrieval and knowledge
SOK-Bench: A Situated Video Reasoning Benchmark with Aligned Open-World Knowledge
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SOK-Bench, a ~44K-question video reasoning benchmark with situated and open-world commonsense knowledge graphs, generated semi-automatically by prompting LLMs/MLLMs, and evaluates video-LLMs and GPT-4V on it.
Problem
Existing video reasoning benchmarks focus on factual or situated reasoning and rarely require broader real-world commonsense knowledge; crowd-sourced annotation of such knowledge is costly and inconsistent.
Approach
An automatic generation pipeline that builds three linked knowledge graphs (situated knowledge graph from video annotations, general knowledge graph from LLM prompting, and situated commonsense knowledge graph via few-shot self-prompting) and uses them to produce multiple-choice and direct-answer QA with rationales.
Key results
- GPT-4V achieves the best overall multiple-choice accuracy of 0.539 on SOK-Bench, while the best open video model (Ask Anything) reaches 0.455.
- Human validation finds 93.08% of generated QA pairs valid.
- Few-shot self-prompting raises the share of concrete situated knowledge from 61% to 97%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| specify an output format in the prompt for GKG generation (ablation) | GKG generation · GKG generation | valid JSON output: 0% -> 11.1% | Helped |
| add examples and an output format to the GKG-generation prompt (ablation) | GKG generation · GKG generation | valid JSON output: 0% -> all outputs | Helped |
| add Few-Shot Self-Prompting examples (ablation) | SCKG generation · SCKG generation | concrete and specific knowledge relations: 61% -> 97% | Helped |
| use MLLMs to provide spatiotemporal information (ablation) | SOK-Bench question generation · Object Counterfactual, Object Contribution, and spatiotemporal variants | questions requiring video observation: 91%, 99%, 94%, and 99% | Helped |
| use API models instead of deployable Vicuna-7B-based models | video QA models · SOK-Bench, multiple-choice | accuracy: 0.264 -> 0.539 | Helped |
Surprising findings
- Blind ChatGPT (text-only) scores 0.365 overall, well above several video models such as Video-LLaMA (0.264), suggesting shortcut cues in the options.
Limitations
- Top-down QA generation is slow and resource-intensive.
- Simpler question types can leak answers through option choices, and GPT-4V struggles on combined spatiotemporal and commonsense questions.
Experimental setup
Datasets: YouCook2 HOMAGE
Benchmarks: SOK-Bench
Metrics: accuracy BLEU BERTScore
Baselines: GPT-4V Video-LLaMA PandaGPT Video-ChatGPT Ask Anything Valley blind ChatGPT
Builds on: VCR STAR Video-ChatGPT
Closest papers by meaning
- SciVideoBench: Benchmarking Scientific Video Reasoning in Large Multimodal Models (ICML 2026 · similarity 0.89)
- OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.88)
- STEP: Enhancing Video-LLMs' Compositional Reasoning by Spatio-Temporal Graph-guided Self-Training (CVPR 2025 · similarity 0.88)
- VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection (CVPR 2025 · similarity 0.88)
- Building a Mind Palace: Structuring Environment-Grounded Semantic Graphs for Effective Long Video Analysis with LLMs (CVPR 2025 · similarity 0.88)
- Black Swan: Abductive and Defeasible Video Reasoning in Unpredictable Events (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.