Research map / Video / Video-language understanding
Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VideoNIAH, a synthetic needle-insertion framework for building skill-targeted video MLLM benchmarks, and uses it to build VNBench (retrieval, ordering, counting) to evaluate 12 video models and analyze training factors.
Problem
Existing real-world video benchmarks are costly to build, prone to data leakage, and entangle multiple capabilities, making them inefficient for skill-specific evaluation during iterative video MLLM development.
Approach
Decouple video content from query-response pairs by inserting unrelated subtitle or image "needles" into original video haystacks, and generate multiple-choice QA automatically from predefined rules targeting temporal perception, chronological ordering, and spatio-temporal coherence.
Key results
- Gemini 1.5 Pro achieves the highest overall VNBench accuracy at 66.7%, while the best open-source model LLaVA-OneVision-72B reaches 58.7%.
- Counting-E2 is hard for all models: GPT-4o scores 0.0% and Gemini 1.5 Pro 7.3%.
- VNBench overall accuracy correlates above 0.9 with VideoMME and MLVU across frame counts (0.91 and 0.95 respectively).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| include frame timestamps in the prompt instead of using bare image tokens (ablation) | video model evaluated on VNBench · VNBench ordering | accuracy: 12.2 -> 21.9 | Helped |
| include frame timestamps in the prompt instead of using bare image tokens (ablation) | video model evaluated on VNBench · VNBench counting | accuracy: 13.3 -> 23.5 | Helped |
| add image index labels to the bare-image prompt (ablation) | video model evaluated on VNBench · VNBench overall | accuracy: 29.3 -> 34.0 | Helped |
| add frame index labels alongside image index labels (ablation) | video model evaluated on VNBench · VNBench overall | accuracy: 34.0 -> 34.7 | Helped |
| increase circular evaluation iterations from one to four (ablation) | Gemini 1.5 Pro · VNBench | overall accuracy: 79.5 -> 66.7 | Hurt |
| increase circular evaluation iterations from one to four (ablation) | GPT-4o · VNBench | overall accuracy: 78.3 -> 64.4 | Hurt |
| replace the original needle content with animal images or new subtitle content (ablation) | VNBench evaluation · VNBench retrieval, ordering, and counting tasks | correlation coefficient: 0.9990; 0.9965; 0.9999; 0.9993; 0.9990 | No clear effect |
| expand the benchmark from 1350 to 2700 samples (ablation) | VNBench evaluation · VNBench overall | correlation coefficient: 0.9996 | No clear effect |
| increase model size (ablation) | LLaVA-OneVision family · VNBench sub-tasks | accuracy | Helped |
| increase tokens per frame using adaptive pooling variants (ablation) | trained video MLLM · VNBench sub-tasks | accuracy | No clear effect |
| increase the number of sampled video frames (ablation) | trained video MLLM · VNBench retrieval and ordering | accuracy | Helped |
| increase the number of sampled video frames (ablation) | trained video MLLM · VNBench counting | accuracy | No clear effect |
Surprising findings
- Open-source models nearly fail the ordering task (most near 0%) while proprietary models reach around 73%.
- Increasing frame token density yields minimal change on VNBench, and the counting task shows only marginal gains from more frames.
Limitations
- Synthetic needles are not real-world content, so the benchmark cannot fully replicate authentic scenarios, as the authors acknowledge.
- Most needles are easy-to-identify objects (subtitles, fruit images), which may limit how far the benchmark measures real video understanding.
- Proprietary model evaluation depends on API versions and sampling settings, and the evaluation cost is not fully reported.
Experimental setup
Datasets: MSRVTT NeXT-QA videos ActivityNet MSCOCO SEED-Bench-2 Something-Something V2 Cauldron VideoChatGPT-100K ShareGPT4Video ShareGPTVideo VIM NExT-QA
Benchmarks: VNBench VNBench-Long VNBench-Act VideoMME MLVU LongVideoBench
Metrics: accuracy circular accuracy Pearson correlation
Baselines: Gemini 1.5 Pro GPT-4o GPT-4-turbo LLaVA-NeXT-Video-7B ST-LLM LLaMA-VID-7B Video-LLaVA-7B VideoChat2 Qwen2-VL-7B LLaVA-OneVision-72B
Builds on: Needle in a Haystack LLaVA-OneVision RULER
Closest papers by meaning
- MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (CVPR 2024 · similarity 0.92)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.92)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.91)
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · similarity 0.91)
- TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning (ICLR 2025 · similarity 0.90)
- VideoReasonBench: Can MLLMs Perform Vision-Centric Complex Video Reasoning? (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.