Research map / Video / Video-language understanding
Two Causally Related Needles in a Video Haystack
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Causal2Needles, a long-video benchmark with noncausal/causal one-needle and causal two-needle questions built from movie recap videos, showing VLMs struggle with joint cause-effect retrieval that degrades with needle distance.
Problem
Existing long-video benchmarks mostly test single-location retrieval (one needle) and rarely test joint understanding of two separate video locations or causal, human-behavior world-modeling; VLMs' ability on these is unclear.
Approach
Two-needle questions are built around a bridge entity shared by a cause and an effect event, so the model must first resolve the vague bridge entity by retrieving the effect clip before retrieving the cause clip; two complementary formats (visual grounding and image description) guard against textual bias and out-of-distribution format effects.
Key results
- On visual grounding 2-needle questions (avg order), ChatGPT-4o reaches 11.4% 'both' accuracy versus 39.2% Part 2 on causal 1-needle for the same cause retrieval, while human both-part accuracy is 79.3%.
- Open-source models mostly score at or near random chance on VG 2-needle questions, e.g. LLaVA-Next-Video-34B 0.0% both-accuracy in avg order.
- Model accuracy on VG 2-needle questions declines as needle distance grows, while human performance stays roughly unchanged (87.5% for distances 6-8 vs 77.3% for 3-5 'both').
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate on causal rather than noncausal 1-needle questions | ChatGPT-4o · Causal2Needles 1-N | accuracy (%): 56.8 -> 39.2 | Hurt |
| Require joint visual grounding of two needles rather than causal 1-needle retrieval | ChatGPT-4o · Causal2Needles VG 2-N Avg Part 2 vs causal 1-N | accuracy (%): 39.2 -> 30.2 | Hurt |
| Use reverse rather than forward clip order | ChatGPT-4o · Causal2Needles VG 2-N Part 1 | accuracy (%): 16.7 -> 45.4 | Helped |
| Use reverse rather than forward clip order | ChatGPT-4o · Causal2Needles VG 2-N Part 2 | accuracy (%): 39.2 -> 21.2 | Hurt |
| Increase the distance between cause and effect clips | VLMs on Causal2Needles VG 2-N · Causal2Needles VG 2-N | accuracy | Hurt |
| Provide video input in addition to narration and question (ablation) | ChatGPT-4o · Causal2Needles causal 1-N | accuracy (%): 3.8 -> 39.2 | Helped |
| Provide video and narration rather than question alone (ablation) | ChatGPT-4o · Causal2Needles ID 2-N | accuracy (%): 47.5 -> 59.2 | Helped |
| Stack five frames from each clip into one image rather than provide non-stacked frames (ablation) | ChatGPT-4o · Causal2Needles VG 2-N Avg Both | accuracy (%): 1.2 -> 11.4 | Helped |
| Stack five frames from each clip into one image rather than provide non-stacked frames (ablation) | Gemini-1.5-pro · Causal2Needles VG 2-N Avg Both | accuracy (%): 4.5 -> 9.3 | Helped |
| Stack five frames from each clip into one image rather than provide non-stacked frames (ablation) | Qwen2.5VL-7B · Causal2Needles VG 2-N Avg Both | accuracy (%): 0.2 -> 3.4 | Helped |
| Use question-only inputs instead of video and narration (ablation) | ChatGPT-4o · Causal2Needles ID 2-N | accuracy (%): 59.2 -> 47.5 | Hurt |
Surprising findings
- Models answer the cause-retrieval Part 2 more accurately than Part 1 in forward order, a positional bias that reverses when the clip order is reversed.
- Some open-source VLMs output the same clip number regardless of question content (static output bias), yielding near-zero two-needle scores.
Limitations
- Videos are movie recaps, which over-represent uncommon dramatic events compared with real life.
- Evaluation covers only visual and text inputs; audio is excluded.
Experimental setup
Datasets: SyMoN YMS
Benchmarks: Causal2Needles
Metrics: accuracy
Baselines: Gemini-1.5-pro ChatGPT-4o ChatGPT-4o-mini Claude-3.5-sonnet Qwen2.5VL-7B Qwen2.5VL-32B LLaVA-Next-Video-7B LLaVA-Next-Video-34B LLaVA-OneVision-7B LongVU-7B Aria-28B LongVA-7B
Builds on: SyMoN YMS event graph extraction
Closest papers by meaning
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.91)
- CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understanding (ICLR 2025 · similarity 0.90)
- Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames (NeurIPS 2025 · similarity 0.89)
- Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs (ICLR 2025 · similarity 0.89)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.88)
- SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.