Research map / Video / Video-language understanding
V^2Dial: Unification of Video and Visual Dialog via Multimodal Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes V2Dial, an expert-based model that unifies video and visual dialog by routing spatial/temporal features through dedicated experts, pre-trained with matching/contrastive losses and coupled with Flan-T5-large, and studies AVSD–VisDial domain shift.
Problem
Video dialog and visual dialog have evolved separately despite structural similarities; existing visual dialog models are trained with NSP heads and ranking, while video dialog models are generative, and their mutual benefit is unexplored.
Approach
Dedicated spatial and temporal experts (plus caption and context language experts) inside multimodal expert layers, trained with spatial-temporal contrastive/matching losses and then coupled to a frozen-then-finetuned LLM under a fully generative objective; VisDial candidates are ranked by embedding cosine similarity to the generated answer.
Key results
- Zero-shot on AVSD-DSTC10, V2Dial more than doubles CIDEr over VideoChat2 (53.2 vs 25.4 is on DSTC8 scale; DSTC10 CIDEr 53.8 vs 25.4 reported in Table 2 row).
- Fine-tuned on VisDial v1.0 with sentence-embedding ranking (OpenAI TE-3), V2Dial reaches R@1 44.9 and MRR 52.4, new state of the art on those metrics.
- Fine-tuned on AVSD-DSTC10, V2Dial reaches CIDEr 103.3, lifting CIDEr by over 6 points over the second-best model (DialogMCF 91.2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use V2Dial instead of VideoChat2 | AVSD-DSTC10 zero-shot · AVSD-DSTC10 | CIDEr: 25.4 -> 53.8 | Helped |
| Use V2Dial instead of MST-MIXER | AVSD-DSTC10 fine-tuning · AVSD-DSTC10 | CIDEr: 96.9 -> 103.3 | Helped |
| Use OpenAI Text Embedding-3large instead of RoBERTa for answer ranking | V2Dial zero-shot · VisDial v1.0 val | R@1: 20.0 -> 22.1 | Helped |
| Use OpenAI Text Embedding-3large instead of RoBERTa for answer ranking | V2Dial fine-tuning · VisDial v1.0 val | R@1: 45.4 -> 46.1 | Helped |
| Remove Stage 1 pre-training (ablation) | V2Dial · AVSD-DSTC7 | CIDEr: 139.8 -> 134.0 | Hurt |
| Remove Stage 2 pre-training (ablation) | V2Dial · AVSD-DSTC7 | CIDEr: 139.8 -> 134.7 | Hurt |
| Remove spatial-temporal contrastive and matching losses (ablation) | V2Dial · AVSD-DSTC7 | CIDEr: 139.8 -> 133.2 | Hurt |
| Replace separate spatial and temporal experts with sequential spatial-temporal attention (ablation) | V2Dial · AVSD-DSTC7 | CIDEr: 139.8 -> 133.8 | Hurt |
| Remove all expert layers (ablation) | V2Dial · VisDial v1.0 val | NDCG: 52.3 -> 42.2 | Hurt |
| Swap spatial and temporal experts (ablation) | V2Dial · AVSD-DSTC7 | CIDEr: 139.8 -> 133.7 | Hurt |
| Swap caption and context experts (ablation) | V2Dial · AVSD-DSTC7 | CIDEr: 139.8 -> 131.1 | Hurt |
| Swap spatial and caption experts (ablation) | V2Dial · AVSD-DSTC7 | CIDEr: 139.8 -> 10.1 | Hurt |
Surprising findings
- Fine-tuning on VisDial degrades AVSD generation metrics because VisDial teaches brief answers while AVSD expects long answers, showing a domain shift between the two tasks.
- Pre-training sequentially on one dataset and then fine-tuning on the other hurts performance on both datasets compared with dataset-specific fine-tuning.
Limitations
- Visual dialog ranking uses an external sentence embedding model rather than a native NSP head, so VisDial retrieval scores depend on the choice of embedder.
- Fully generative training underperforms discriminative NSP-based models on several VisDial retrieval metrics (e.g., MRR, R@5/R@10 in fine-tuning).
Experimental setup
Datasets: WebVid-2M CC-3M Champagne AVSD VisDial v1.0
Benchmarks: AVSD-DSTC7 AVSD-DSTC8 AVSD-DSTC10 VisDial v1.0
Metrics: BLEU-1 BLEU-2 BLEU-3 BLEU-4 METEOR ROUGE-L CIDEr R@1 R@5 R@10
Baselines: MoE-LLaVA MiniGPT4-video Video-ChatGPT MST-MIXER VideoChat2 FROMAGe Champagne ESPER PDC THAM DialogMCF VideoLLaMA 2
Builds on: BLIP-2 Flan-T5 EVA-CLIP ALBEF
Closest papers by meaning
- Structure-Aware Multimodal Sequential Learning for Visual Dialog (AAAI 2024 · similarity 0.91)
- MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs (NeurIPS 2024 · similarity 0.89)
- Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding (CVPR 2024 · similarity 0.88)
- F2RVLM: Boosting Fine-grained Fragment Retrieval for Multi-Modal Long-form Dialogue with Vision Language Model (AAAI 2026 · similarity 0.88)
- SAMA: Towards Multi-Turn Referential Grounded Video Chat with Large Language Models (NeurIPS 2025 · similarity 0.88)
- VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.