Research map / Video / Video-language understanding
Logic-in-Frames: Dynamic Keyframe Search via Visual Semantic-Logical Verification for Long Video Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VSLS, a training-free keyframe search that uses YOLO-World detections and four logic relations (spatial, temporal, attribute, causal) to iteratively update a frame sampling distribution for long-video QA.
Problem
Selecting query-relevant keyframes from long videos under tight frame budgets, where existing captioning, clustering, and retrieval methods ignore logical relations between query and visual content.
Approach
Parse the question into key/cue objects and four logical relations (spatial co-occurrence, temporal proximity, attribute dependency, causal order), then iteratively sample frames, score them with a lightweight open-vocabulary detector, boost scores by relation satisfaction, and diffuse scores to neighboring frames to refine the sampling distribution.
Key results
- On LongVideoBench with 32 frames, VSLS reaches 74.5 precision, 92.5 recall, and 41.4 temporal coverage, above T* (36.5 TC).
- With GPT-4o at 8 frames on LongVideoBench long videos, VSLS raises accuracy from 47.1 to 51.2 (about +4 points).
- On LV-Haystack, VSLS-Detector achieves 61.5% overall accuracy versus 59.8% for T*-Detector at a similar latency (7.8 s search).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add semantic-logical search to the detector-based temporal search | T*-Detector · LV-Haystack | Acc: 59.8 -> 61.5 | Helped |
| add VSLS keyframe search | LongVideoBench 8-frame retrieval · LongVideoBench 8 frames | Recall: 88.2 -> 88.6 | Helped |
| add VSLS keyframe search | LongVideoBench 32-frame retrieval · LongVideoBench 32 frames | TC: 36.5 -> 41.4 | Helped |
| add VSLS keyframe search | GPT-4o · LongVideoBench long videos, 8 frames | accuracy: 47.1 -> 51.2 | Helped |
| add VSLS keyframe search | GPT-4o · Video-MME long videos, 8 frames | accuracy: 55.2 -> 56.9 | Helped |
| add VSLS keyframe search | GPT-4o · LongVideoBench long videos, 32 frames | accuracy: 53.8 -> 54.2 | Helped |
| add VSLS instead of T* | GPT-4o · LongVideoBench long videos, 32 frames | accuracy: 55.3 -> 54.2 | Hurt |
| add the spatial relation detector (ablation) | T* on spatial-relation LongVideoBench queries · LongVideoBench spatial relation | TC: 37.5 -> 45.5 | Helped |
| add the attribute relation detector (ablation) | T* on attribute-relation LongVideoBench queries · LongVideoBench attribute relation | TC: 38.5 -> 42.1 | Helped |
| add the time relation detector (ablation) | T* on time-relation LongVideoBench queries · LongVideoBench time relation | TC: 37.3 -> 36.1 | Hurt |
| add the causal relation detector (ablation) | T* on causal-relation LongVideoBench queries · LongVideoBench causal relation | TC: 38.6 -> 39.6 | Helped |
Surprising findings
- The paper reports that spatial relations are the most frequently detected logic type, while the temporal relation gave a slight drop in temporal coverage.
Limitations
- Relies on a lightweight open-vocabulary detector (YOLO-World) and LLM/VLM query parsing, so detection errors propagate to the search.
- Relation hyperparameters (alpha, gamma, tau, Delta_t) are tuned by grid search on the evaluation datasets.
- Results on Video-MME are mixed; gains are not uniform across models and video lengths.
Experimental setup
Datasets: LongVideoBench Video-MME LV-Haystack Ego4D
Benchmarks: LongVideoBench Video-MME LV-Haystack
Metrics: precision (SSIM) recall (SSIM) temporal coverage accuracy TFLOPs latency
Baselines: Uniform sampling VideoAgent T*-Retrieval T*-Attention T*-Detector GPT-4o + T*
Builds on: T* YOLO-World VideoAgent
Closest papers by meaning
- Re-thinking Temporal Search for Long-Form Video Understanding (CVPR 2025 · similarity 0.91)
- Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames (NeurIPS 2025 · similarity 0.90)
- Adaptive Keyframe Sampling for Long Video Understanding (CVPR 2025 · similarity 0.90)
- Not Another Text Benchmark: Putting the “Visual" Back in Visual Question Answering for Large Video Models (NeurIPS 2026 · similarity 0.89)
- VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection (CVPR 2025 · similarity 0.89)
- AdaVideoRAG: Omni-Contextual Adaptive Retrieval-Augmented Efficient Long Video Understanding (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.