Research map / Video / Video-language understanding
A Video-grounded Dialogue Dataset and Metric for Event-driven Activities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VDAct, a video-grounded dialogue dataset of 3,000 dialogues over 1,000 VirtualHome scenario videos with knowledge graphs, and VDEval, an LLM-based session-context metric that correlates better with humans than turn-based metrics.
Problem
Existing video-grounded dialogue datasets use short, simple activity videos with mostly descriptive questions, and existing text/QA metrics compare responses only to a single reference, ignoring dialogue history and video context that make multiple responses valid.
Approach
Builds a dialogue dataset on multi-activity VirtualHome scenario videos paired with scenario knowledge graphs and summaries, and proposes VDEval, an LLM judge that uses full session dialogue history plus a KG-derived video summary instead of single-turn context (extending LAVE).
Key results
- Session-based VDEval with refined KG summaries reaches Kendall-B correlation 73.62 with human judgments on fine-tuned VideoLLaMA2, versus 70.08 for the best existing metric (LLM-Rel).
- All baseline models score below 50% on VDEval; fine-tuned VideoLLaMA2 (8 frames) is best open model, and GPT-4o (16 frames) reaches 41.26 VDEval.
- Increasing VideoLLaMA2 input from 8 to 16 frames decreases performance across all metrics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use session-based rather than turn-based context (ablation) | VDEval · Human judgment correlation, VL2 frozen | Kendall's rank correlation: 66.85 -> 68.25 | Helped |
| Use session-based rather than turn-based context (ablation) | VDEval · Human judgment correlation, VL2 finetuned | Kendall's rank correlation: 70.09 -> 72.59 | Helped |
| Add a refined KG summary (ablation) | VDEval with session-based context · Human judgment correlation, VL2 frozen | Kendall's rank correlation: 68.25 -> 72.30 | Helped |
| Add a refined KG summary (ablation) | VDEval with session-based context · Human judgment correlation, VL2 finetuned | Kendall's rank correlation: 72.59 -> 73.62 | Helped |
| Replace a template KG summary with a refined KG summary (ablation) | VDEval with session-based context · Human judgment correlation, VL2 frozen | Kendall's rank correlation: 69.12 -> 72.30 | Helped |
| Replace a template KG summary with a refined KG summary (ablation) | VDEval with session-based context · Human judgment correlation, VL2 finetuned | Kendall's rank correlation: 71.63 -> 73.62 | Helped |
| Increase VideoLLaMA2 input frames from 8 to 16 (ablation) | VideoLLaMA2, LoRA fine-tuned · VDAct test set | VDEval: 43.13 -> 42.39 | Hurt |
Surprising findings
- Increasing the number of video frames from 8 to 16 for VideoLLaMA2 hurt performance on all metrics.
- Template-based KG summaries reduced correlation for the fine-tuned VideoLLaMA2 under session-based context, while refined summaries consistently helped.
Limitations
- Videos are synthetic VirtualHome simulations with a single agent, limiting realism.
- Human evaluation covers only 20% of the test set (90 dialogues) with three annotators.
- The dataset is smaller in number of videos and dialogues than existing datasets such as AVSD.
Experimental setup
Datasets: VDAct VirtualHome2KG
Benchmarks: VDAct
Metrics: BLEU ROUGE METEOR SPICE LLM-Acc LLM-Rel LAVE VDEval Kendall-B correlation with human judgments
Baselines: Video-LLaVA Video-ChatGPT VideoLLaMA2 GPT-4o Gemini-1.5-pro BLEU ROUGE METEOR SPICE BERTScore BARTScore LLM-Acc
Builds on: LAVE VirtualHome2KG AVSD
Closest papers by meaning
- E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding (NeurIPS 2024 · similarity 0.85)
- SAMA: Towards Multi-Turn Referential Grounded Video Chat with Large Language Models (NeurIPS 2025 · similarity 0.84)
- HourVideo: 1-Hour Video-Language Understanding (NeurIPS 2024 · similarity 0.84)
- V^2Dial: Unification of Video and Visual Dialog via Multimodal Experts (CVPR 2025 · similarity 0.84)
- VLog: Video-Language Models by Generative Retrieval of Narration Vocabulary (CVPR 2025 · similarity 0.84)
- Unleashing Hour-Scale Video Training for Long Video-Language Understanding (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.