Research map / Video / Video-language understanding
HierarQ: Task-Aware Hierarchical Q-Former for Enhanced Video Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HierarQ, a task-aware hierarchical Q-Former with entity- and prompt-guided feature modulators and short/long-term memory banks that sequentially processes video frames for medium-to-long video understanding, QA and captioning.
Problem
Medium-to-long video understanding in MLLMs is limited by LLM context length and frame sampling, which loses key temporal information and lacks task-specific relevance.
Approach
Process every frame sequentially through a two-stream language-guided feature modulator (entity stream and scene stream, using BERT text embeddings and cross-attention) feeding a hierarchical Q-Former whose two levels each have dedicated memory banks (FIFO for short-term entity memory, memory bank compression for long-term scene memory), so the LLM receives only 32 tokens per step.
Key results
- On LVU, HierarQ reaches 67.9% average top-1 accuracy, surpassing MA-LMM by 6.8 points.
- On MovieChat-1k global mode, HierarQ achieves 87.5% GPT-3.5-assisted accuracy vs. 84.0 for FVS+S3.
- On MSVD-QA, HierarQ reaches 66.2% accuracy, and 80.5 CIDEr on MSRVTT captioning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the HierarQ framework | Vicuna 7B video understanding · LVU | Avg accuracy: 61.1 -> 67.9 | Helped |
| Add the HierarQ framework | Vicuna 7B video understanding · Breakfast | top-1 accuracy: 94.3 -> 97.4 | Helped |
| Add the HierarQ framework | Vicuna 7B video understanding · COIN | top-1 accuracy: 93.2 -> 96.0 | Helped |
| Add the HierarQ framework | Vicuna 7B video QA · MovieChat-1k Global | GPT-3.5-assisted accuracy: 84.0 -> 87.5 | Helped |
| Add the HierarQ framework | Vicuna 7B video QA · MSRVTT-QA | top-1 accuracy: 50.4 -> 54.1 | Helped |
| Add the HierarQ framework | Vicuna 7B video QA · MSVD-QA | top-1 accuracy: 60.6 -> 66.2 | Helped |
| Add the HierarQ framework | Vicuna 7B video QA · ActivityNet-QA | top-1 accuracy: 51.1 -> 57.1 | Helped |
| Add the HierarQ framework | Vicuna 7B video captioning · MSRVTT | CIDEr: 80.3 -> 80.5 | Helped |
| Use the prompt-guided feature modulator alone instead of the baseline (ablation) | HierarQ · LVU | accuracy: 60.7 -> 62.0 | Helped |
| Use the entity-guided feature modulator alone instead of the baseline (ablation) | HierarQ · LVU | accuracy: 60.7 -> 58.7 | Hurt |
| Do not finetune the LLM (ablation) | HierarQ · LVU | accuracy: 67.9 -> 66.8 | Hurt |
| Remove short- and long-term memory banks (ablation) | HierarQ · LVU | accuracy: 55.8 -> 67.9 | Hurt |
Surprising findings
- Using the entity-guided modulator alone slightly decreases performance relative to the baseline, attributed to limited scene context.
Limitations
- Entity-only modulation alone hurts performance, indicating dependence on scene context.
- Short-term memory performance degrades beyond length 10, suggesting excess entity information can overshadow scene context.
Experimental setup
Datasets: LVU Breakfast COIN MovieChat-1k MSRVTT-QA MSVD-QA ActivityNet-QA MSRVTT MSVD YouCook2
Benchmarks: LVU Breakfast COIN MovieChat-1k MSRVTT-QA MSVD-QA ActivityNet-QA MSRVTT MSVD YouCook2
Metrics: top-1 accuracy GPT-3.5-assisted accuracy GPT-3.5-assisted score exact-match accuracy CIDEr
Baselines: MA-LMM S5 Movies2Scene VideoMamba FVS+S3 MovieChat GIT mPLUG-2
Builds on: BLIP-2 Q-Former MA-LMM InstructBLIP MovieChat
Closest papers by meaning
- AdaVideoRAG: Omni-Contextual Adaptive Retrieval-Augmented Efficient Long Video Understanding (NeurIPS 2025 · similarity 0.88)
- Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.88)
- ReWind: Understanding Long Videos with Instructed Learnable Memory (CVPR 2025 · similarity 0.88)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.88)
- Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames (NeurIPS 2025 · similarity 0.88)
- VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.