Research map / Large language models / Agents, code and math
Enhancing Video-LLM Reasoning via Agent-of-Thoughts Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills LLM-verified chain-of-thoughts, generated by an agent system of specialized vision tools decomposing VideoQA questions, into LLaVA-NeXT-Video via instruction tuning to improve reasoning and VideoQA accuracy.
Problem
Video-LLMs trained only on question-answer pairs lack explainability and spatial-temporal grounding, while agent-based VideoQA systems are interpretable but weak and slow.
Approach
Automatically construct step-by-step CoTs by executing agent-generated programs over specialized vision models on existing QA datasets, filter them with an LLM verifier, and distill them into a Video-LLM alongside answers, using a suffix prompt to switch between answer-only and rationale output.
Key results
- LLaVA-NeXT-Video-AoTD reaches 74.3% on STAR versus 72.2% for the instruction-tuned LLaVA-NeXT-Video-Instruct, and 55.6% on MVBench versus 53.4%.
- On open-ended AGQA, AoTD scores 60.9 accuracy versus 59.3 for the Instruct variant; on ActivityNet-QA it reaches 53.2 accuracy-level vs 50.0.
- Removing LLM CoT filtering lowers STAR and MVBench performance (MVBench 53.7 vs 55.6 with filtering).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| distill verified CoTs instead of QA-only instruction tuning | LLaVA-NeXT-Video (7B) · MVBench | accuracy: 53.4 -> 55.6 | Helped |
| distill verified CoTs instead of QA-only instruction tuning | LLaVA-NeXT-Video (7B) · VideoMME | accuracy: 43.2 -> 45.0 | Helped |
| distill verified CoTs instead of QA-only instruction tuning | LLaVA-NeXT-Video (7B) · STAR | accuracy: 72.2 -> 74.3 | Helped |
| distill verified CoTs instead of QA-only instruction tuning | LLaVA-NeXT-Video (7B) · NExT-QA | accuracy: 77.1 -> 77.6 | Helped |
| distill verified CoTs instead of QA-only instruction tuning | LLaVA-NeXT-Video (7B) · Perception-Test | accuracy: 50.3 -> 50.6 | Helped |
| distill verified CoTs instead of QA-only instruction tuning | LLaVA-NeXT-Video (7B) · VSIBench | accuracy: 26.7 -> 28.8 | Helped |
| distill verified CoTs instead of QA-only instruction tuning | LLaVA-NeXT-Video (7B) · ANetQA | accuracy: 47.1 -> 53.9 | Helped |
| distill verified CoTs instead of QA-only instruction tuning | LLaVA-NeXT-Video (7B) · AGQA | accuracy: 59.3 -> 60.9 | Helped |
| distill verified CoTs instead of QA-only instruction tuning | LLaVA-NeXT-Video (7B) · ActivityNet-QA | accuracy: 50.0 -> 53.2 | Helped |
| remove CoT filtering (ablation) | LLaVA-NeXT-Video-AoTD · MVBench | accuracy: 55.6 -> 53.7 | Hurt |
| remove CoT filtering (ablation) | LLaVA-NeXT-Video-AoTD · STAR | accuracy: 74.3 -> 73.3 | Hurt |
| remove CoT filtering (ablation) | LLaVA-NeXT-Video-AoTD · AGQA | accuracy: 60.9 -> 59.5 | Hurt |
Surprising findings
- Instruction-tuned LLaVA-NeXT-Video without CoTs struggles to generate valid rationales, while the AoTD model's temporal IoU is comparable to UniVTG.
Limitations
- Current video understanding tools are not strong enough for reliable agent systems, and agent systems are memory-heavy and slow.
- Open-ended evaluation relies on GPT-assessed metrics that may be biased or inaccurate.
- Distilled model does not surpass LLaVA-NeXT-Video on some open-ended benchmarks.
Experimental setup
Datasets: STAR NExT-QA AGQA ANetQA CLEVRER EgoQA
Benchmarks: STAR NExT-QA MVBench VideoMME Perception-Test VSIBench AGQA ActivityNet-QA ANetQA Video-ChatGPT
Metrics: accuracy GPT-assessed accuracy GPT-assessed score IoU recall
Baselines: LLaVA-NeXT-Video LLaVA-NeXT-Video-Instruct VideoChat2 VideoLLaMA2 LLaMA-VID Video-LLaVA Gemini 1.5 Pro GPT4-V LLaVA-OneVision
Builds on: STAR LLaVA-NeXT-Video Visual Program Distillation Fact Video-STaR
Closest papers by meaning
- Divide and Conquer: Exploring Language-centric Tree Reasoning for Video Question-Answering (ICML 2025 · similarity 0.90)
- VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection (CVPR 2025 · similarity 0.90)
- MoReVQA: Exploring Modular Reasoning Models for Video Question Answering (CVPR 2024 · similarity 0.89)
- Modularized Self-Reflected Video Reasoner for Multimodal LLM with Application to Video Question Answering (ICML 2025 · similarity 0.89)
- STEP: Enhancing Video-LLMs' Compositional Reasoning by Spatio-Temporal Graph-guided Self-Training (CVPR 2025 · similarity 0.89)
- Language-aware Visual Semantic Distillation for Video Question Answering (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.