Research map / Video / Video-language understanding
VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VideoTree, a training-free framework that builds a query-adaptive, coarse-to-fine tree of keyframe captions via iterative clustering and LLM relevance scoring, then answers with GPT-4 for long-video QA.
Problem
Long-form video question answering suffers from high redundancy and query-irrelevant information when videos are converted to text captions for LLM reasoning, and flat caption lists miss the hierarchical structure of video.
Approach
Build the video representation top-down: adaptively cluster frames and score cluster-caption relevance with an LLM, then recursively sub-cluster highly relevant clusters into a tree so that more keyframes are allocated to query-relevant regions before LLM reasoning.
Key results
- Achieves 66.2% on EgoSchema subset and 61.1% on full test set, and 75.6% average on NExT-QA with GPT-4, outperforming prior training-free methods such as LLoVi (61.2 subset) and VideoAgent (60.2 subset).
- On the long split of Video-MME (avg 44 min), reaches 54.2% accuracy, beating GPT-4V (53.5%) and LLoVi (48.8%).
- With open-source Mistral 7B/12B and log-likelihood classifier, improves EgoSchema subset accuracy over LangRepo by 2.2% and 4.8% while using about 72.5% and 69.0% less inference time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add depth expansion (ablation) | VideoTree · EgoSchema subset | accuracy: 64.4 -> 66.2; 1.8% | Helped |
| add adaptive breadth expansion (ablation) | VideoTree · EgoSchema subset | accuracy: 61.2 -> 66.2; 3.2% | Helped |
| use fewer frames by changing hyperparameters (VideoTree-fast) (ablation) | VideoTree · EgoSchema subset | accuracy: 57.8; 61.2; 63.6; 66.2 | Mixed |
| use fewer frames by changing hyperparameters (VideoTree-fast) (ablation) | VideoTree · EgoSchema subset | inference time: 7.8; 20.1; 33% the time cost | Helped |
| use Mistral-7B as the LLM reasoner with a log-likelihood classifier (ablation) | VideoTree · EgoSchema subset | accuracy: 60.8; 63.0; 2.2% | Helped |
| use Mistral-8×7B (12B) as the LLM reasoner with a log-likelihood classifier (ablation) | VideoTree · EgoSchema subset | accuracy: 66.2; 71.0; 4.8% | Helped |
| use a hierarchical representation that can generalize to more frame captions (ablation) | VideoTree · EgoSchema subset | accuracy: 6% boost at its peak | Helped |
| use VideoTree instead of LLoVi | GPT-4-based training-free video QA · Video-MME long split | accuracy: 5.4% improvement | Helped |
| use VideoTree instead of GPT-4V | Video QA on Video-MME long split · Video-MME long split | accuracy: 0.7% | Helped |
Surprising findings
- Removing adaptive breadth expansion causes a larger drop (3.2%) than removing depth expansion (1.8%) on EgoSchema subset.
Limitations
- Performance depends on the captioner and strong proprietary LLM; open-source LLM results lag substantially behind GPT-4.
- Gap remains to long-context proprietary MLLMs such as GPT-4o and Gemini 1.5 Pro on Video-MME long split.
Experimental setup
Datasets: EgoSchema NExT-QA Video-MME
Benchmarks: EgoSchema NExT-QA Video-MME
Metrics: accuracy
Baselines: LLoVi VideoAgent LangRepo LifelongMemory IG-VLM LVNet GPT-4V Gemini 1.5 Pro GPT-4o MVU Video-LLaVA+INTP
Builds on: LLoVi VideoAgent LangRepo K-Means clustering
Closest papers by meaning
- Vgent: Graph-based Retrieval-Reasoning-Augmented Generation For Long Video Understanding (NeurIPS 2025 · similarity 0.90)
- SF2T: Self-supervised Fragment Finetuning of Video-LLMs for Fine-Grained Understanding (CVPR 2025 · similarity 0.89)
- VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection (CVPR 2025 · similarity 0.88)
- ALLVB: All-in-One Long Video Understanding Benchmark (AAAI 2025 · similarity 0.88)
- Temporal Reasoning Transfer from Text to Video (ICLR 2025 · similarity 0.88)
- SlowFocus: Enhancing Fine-grained Temporal Understanding in Video LLM (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.