Research map / Video / Video-language understanding
Granularity-Adaptive Spatial Evidence Tokenization for Video Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a granularity-adaptive spatial evidence tokenization model for video QA that highlights question-relevant patches at low resolution, encodes only those patches at high resolution with masking acceleration, and compresses multi-granular tokens by question.
Problem
Video question answering models encode frames at fixed low resolution, losing fine-grained spatial detail, while existing high-resolution image LLMs encode entire high-res frames at high computational cost.
Approach
Use question-guided spatial activation at low resolution to select patches, map them to high-resolution frames via index querying, encode only the masked activated patches (with relative position embeddings), and select/compress tokens across granularities with a question-conditioned granularity compression module.
Key results
- Achieves 73.4% accuracy on MSVD-QA and 51.4% on ActivityNet-QA, outperforming listed 7B baselines in open-ended QA (Table 1).
- Reaches 45.1% average on MVBench, above VideoLLaVA (42.9) and LLaMA-VID (42.2) but below VideoChat2 (51.1) which uses far more instruction data.
- Reduces model complexity versus LLaMA-VID from 16.84 to 11.97 TMACs and 33.70 to 23.94 TFLOPs (Table 8).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add granularity weighting module (ablation) | Granularity-adaptive tokenization model · MSVD-QA | accuracy: 72.0 -> 73.4 | Helped |
| add granularity compression module (ablation) | Granularity-adaptive tokenization model · MSVD-QA | accuracy: 72.5 -> 73.4 | Helped |
| add multi-granularity tokenization module (ablation) | Granularity-adaptive tokenization model · MSVD-QA | accuracy: 71.9 -> 73.4 | Helped |
| add granularity weighting module (ablation) | Granularity-adaptive tokenization model · ActivityNet-QA | accuracy: 49.7 -> 51.4 | Helped |
| add granularity compression module (ablation) | Granularity-adaptive tokenization model · ActivityNet-QA | accuracy: 49.7 -> 51.4 | Helped |
| add multi-granularity tokenization module (ablation) | Granularity-adaptive tokenization model · ActivityNet-QA | accuracy: 49.8 -> 51.4 | Helped |
| add granularity weighting module (ablation) | Granularity-adaptive tokenization model · TGIF-QA | accuracy: 72.4 -> 74.9 | Helped |
| add granularity compression module (ablation) | Granularity-adaptive tokenization model · TGIF-QA | accuracy: 73.8 -> 74.9 | Helped |
| add multi-granularity tokenization module (ablation) | Granularity-adaptive tokenization model · TGIF-QA | accuracy: 74.0 -> 74.9 | Helped |
| use the proposed masking and acceleration strategy instead of encoding all high-resolution frame patches (ablation) | Granularity-adaptive tokenization model · ActivityNet-QA | TFLOPs: 53.40 -> 23.94 | Helped |
| use prompt PR #2 instead of PR #1 | Granularity-adaptive tokenization model · NExT-QA | accuracy: 64.0 -> 61.8 | Mixed |
| use system instruction INS #2 instead of INS #1 | Granularity-adaptive tokenization model · NExT-QA | accuracy: 62.8 -> 64.1 | Helped |
Surprising findings
- Model performance varies only slightly across different question prompts and system instructions, per the prompt ablation.
Limitations
- Relies on a pretrained frozen visual encoder and Q-Former, so resolution gains depend on these components.
- Evaluation relies on GPT-3.5 as judge for open-ended QA and text generation.
- Comparisons with some baselines (VideoChat2, VideoLLaMA2) are excluded or uneven due to differing training data scale.
Experimental setup
Datasets: MSVD-QA MSRVTT-QA ActivityNet-QA TGIF-QA Video-ChatGPT text generation benchmark NExT-QA STAR TVQA IntentQA EgoSchema MVBench
Benchmarks: MSVD-QA MSRVTT-QA ActivityNet-QA TGIF-QA Video-ChatGPT text generation benchmark NExT-QA STAR TVQA IntentQA EgoSchema MVBench
Metrics: accuracy score
Baselines: Video-LLaMA LLaMA-Adapter Video-ChatGPT Video-LLaVA LLaMA-VID Chat-UniVi VideoChat2 VideoLLaMA2 Monkey MiniGPT4-V
Builds on: LLaMA-VID Monkey InstructBLIP BLIP-2
Closest papers by meaning
- Language-aware Visual Semantic Distillation for Video Question Answering (CVPR 2024 · similarity 0.89)
- Tool-Augmented Spatiotemporal Reasoning for Streamlining Video Question Answering Task (NeurIPS 2025 · similarity 0.88)
- VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection (CVPR 2025 · similarity 0.88)
- Video-Panda: Parameter-efficient Alignment for Encoder-free Video-Language Models (CVPR 2025 · similarity 0.87)
- Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames (NeurIPS 2025 · similarity 0.86)
- Not Another Text Benchmark: Putting the “Visual" Back in Visual Question Answering for Large Video Models (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.