Research map / Video / Video-language understanding
DynFocus: Dynamic Cooperative Network Empowers LLMs with Video Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DynFocus, a dynamic cooperative connector that selects answer-relevant event prototypes via a differentiable DPC-KNN-based selector and encodes them with fine tokens while compressing the rest into few tokens, for memory-efficient video LLMs.
Problem
Long-video understanding with LLMs must preserve visual and semantic detail while keeping the visual token count memory-affordable; redundant and question-dependent frames make fixed encoding wasteful.
Approach
A Dynamic Event Prototype Estimation module uses DPC-KNN clustering plus a learned score net trained end-to-end through a perturbed-maximum differentiable top-K, and a Compact Cooperative Encoding module encodes selected prototypes with many tokens (cone-like) and the rest with two tokens (rod-like).
Key results
- On LV-Bench, DynFocus† (L=70, K/L=0.4) reaches 32.9% overall, the best among open-source 7B models and above PLLaVA-34B (26.1%).
- On VideoMME without subtitles, DynFocus reaches 44.1% overall accuracy, above ST-LLM with subtitles (42.3%).
- On MSVD-QA, DynFocus (Turbo-0613) reaches 74.8% accuracy, comparable to ST-LLM's 74.6%, while using about 25% fewer tokens than ST-LLM on ANet-QA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DynFocus instead of ST-LLM | Vicuna-7B-1.5 video QA · MSVD-QA | accuracy: 74.6 -> 74.8 | Helped |
| Use DynFocus instead of ST-LLM | Vicuna-7B-1.5 video QA · MSRVTT-QA | accuracy: 63.2 -> 62.8 | Hurt |
| Use DynFocus instead of ST-LLM | Vicuna-7B-1.5 video QA · ANet-QA | accuracy: 50.9 -> 50.3 | Hurt |
| Post-tune DynFocus on VideoChatGPT-100K (ablation) | DynFocus · VCG-Bench | Avg.: 3.05 -> 3.17 | Helped |
| Use DynFocus instead of Gemini 1.5 Pro | video understanding · LV-Bench | Overall: 33.1 -> 32.9 | Hurt |
| Increase input from 16 to 32 frames | DynFocus · MLVU | M-Avg: 48.3 -> 49.6 | Helped |
| Use DynFocus instead of VILA-1.5 | video understanding without subtitles · VideoMME | Overall accuracy (%): 59.0 -> 44.1 | Hurt |
| Use DynFocus instead of VILA-1.5 | video understanding with subtitles · VideoMME | Overall accuracy (%): 59.4 -> 47.8 | Hurt |
| Add Rods encoding tokens for marginal frames (ablation) | DynFocus · MSVD-QA | accuracy: 63.7 -> 68.4 | Helped |
| Add Cones encoding tokens for important frames (ablation) | DynFocus · MSVD-QA | accuracy: 58.2 -> 68.4 | Helped |
| Add Rods encoding tokens for marginal frames (ablation) | DynFocus · ANet-QA | accuracy: 38.6 -> 43.1 | Helped |
| Add Cones encoding tokens for important frames (ablation) | DynFocus · ANet-QA | accuracy: 38.6 -> 43.1 | Helped |
Surprising findings
- Using all tokens without compression for important frames (256 tokens) performs only marginally better than 40-token compression on MSVD-QA, and degrades at scale.
- Increasing the number of initial event prototypes beyond 25 hurts performance, which the authors attribute to disrupted temporal structure.
Limitations
- Weaker on ego-centric tasks such as ER on MLVU, attributed to lack of ego-centric training data.
- Lags ST-LLM on temporal understanding in VCG-Bench, possibly because it retains fewer tokens per frame.
Experimental setup
Datasets: LLaVA-filter-CC3M WebVid-2.5M LLaVA-665K ScienceQA VideoChatGPT-100K WebVid-10M-QA NExT-QA CLEVRER MSVD-QA MSRVTT-QA ANet-QA VCG-Bench
Benchmarks: MSVD-QA MSRVTT-QA ANet-QA VCG-Bench MLVU LVBench VideoMME VideoHallucer
Metrics: accuracy score M-Avg G-Avg
Baselines: ST-LLM VideoChat2 LLaMA-VID Chat-UniVi PLLaVA MA-LMM MovieChat Gemini 1.5 Pro GPT-4o
Builds on: LLaMA-VID Chat-UniVi DPC-KNN perturbed maximum method InstructBLIP
Closest papers by meaning
- Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.89)
- SlowFocus: Enhancing Fine-grained Temporal Understanding in Video LLM (NeurIPS 2024 · similarity 0.88)
- ReWind: Understanding Long Videos with Instructed Learnable Memory (CVPR 2025 · similarity 0.88)
- Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders (NeurIPS 2025 · similarity 0.88)
- Recurrent Attention-based Token Selection for Efficient Streaming Video-LLMs (NeurIPS 2025 · similarity 0.87)
- Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.