Research map / Video / Video-language understanding
Re-thinking Temporal Search for Long-Form Video Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Frames temporal search for long videos as a "Long Video Haystack" problem, introduces the LV-Haystack benchmark (15,092 annotated instances) and T*, a lightweight search method that recasts temporal search as spatial search over image grids with iterative zoom-in.
Problem
Selecting a minimal set of relevant keyframes from tens of thousands of frames for long-form video question answering is costly and unevaluated for temporal search capability in long-context VLMs.
Approach
Reframe temporal search as spatial search: tile sampled frames into a grid image, use an object detector or VLM attention to score cells for target/cue objects, and iteratively resample and zoom into high-scoring temporal regions with a spline-updated sampling distribution.
Key results
- On LongVideoBench XLong subset with 32 frames, T* raises LLaVA-OneVision-72B from 56.5% to 62.4% and GPT-4o from 50.5% to 53.1%.
- On LV-Haystack, current SOTA search methods reach only about 2.1% temporal F1 on the LongVideoBench subset.
- Detector-based T* reaches 3.4% temporal F1 on HAYSTACK-EGO4D with 32 frames (vs 2.0% for uniform) and uses about 3x fewer FLOPs than VideoAgent-style search (33.3 vs 536.5 TFLOPs search cost).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add detector-based T* with 32-frame search | temporal search · HAYSTACK-EGO4D | temporal F1: 2.0 -> 3.4 | Helped |
| Add detector-based T* with 32-frame search | temporal search · HAYSTACK-LVBENCH | visual F1: 67.3 -> 67.8 | Helped |
| Add T* frame search with a 32-frame budget | GPT-4o · LongVideoBench XLong | accuracy: 50.5 -> 53.1 | Helped |
| Add T* frame search with a 32-frame budget | LLaVA-OneVision-72B · LongVideoBench XLong | accuracy: 56.5 -> 62.4 | Helped |
| Add T* frame search with a 32-frame budget | GPT-4o · Video-MME Total | accuracy: 61.8 -> 64.1 | Helped |
| Add T* frame search with a 32-frame budget | LLaVA-OneVision-72B · Video-MME Total | accuracy: 66.3 -> 68.3 | Helped |
| Add detector-based T* frame search | LLaVA-OneVision-7B · NExT-QA | accuracy: 76.4 -> 80.4 | Helped |
| Add detector-based T* frame search | LLaVA-OneVision-7B · EgoSchema | accuracy: 63.6 -> 66.6 | Helped |
| Add T* frame search with 32 frames | GPT-4o · Ego4D LongVideo QA Tiny Video | accuracy: 43.2 -> 49.5 | Helped |
| Add T* frame search with 16 frames | Qwen2.5-VL-72B · Ego4D LongVideo QA Test Video | accuracy: 21.0 -> 25.5 | Helped |
Surprising findings
- Current SOTA temporal search methods achieve only around 2% temporal F1, indicating a large gap in search capability.
- Increasing searched frames improves recall but reduces precision for retrieval-based methods.
Limitations
- Assumes most questions can be answered with a few keyframes, which may not hold for tasks needing broader or dense context.
- Focuses on visual cues only, without audio or subtitles.
Experimental setup
Datasets: Ego4D NLQ LongVideoBench Video-MME EgoSchema NExT-QA
Benchmarks: LV-Haystack LongVideoBench Video-MME NExT-QA EgoSchema Ego4D LongVideo QA
Metrics: temporal precision temporal recall temporal F1 visual F1 accuracy TFLOPs latency
Baselines: Uniform sampling VideoAgent Retrieval-based search GPT-4o LLaVA-OneVision-72B LVNet VideoTree
Builds on: V* VideoAgent LongVideoBench YOLO-World
Closest papers by meaning
- Logic-in-Frames: Dynamic Keyframe Search via Visual Semantic-Logical Verification for Long Video Understanding (NeurIPS 2025 · similarity 0.91)
- Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames (NeurIPS 2025 · similarity 0.89)
- ReWind: Understanding Long Videos with Instructed Learnable Memory (CVPR 2025 · similarity 0.89)
- MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughly (NeurIPS 2025 · similarity 0.89)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.89)
- Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.