Research map / Vision-language and multimodal / Human and embodied modeling
Structure-Aware Multimodal Sequential Learning for Visual Dialog
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a structure-aware cross-modulation method that aligns frozen ViT and Flan-T5 via spatiotemporal-graph-aware cross-attention blocks for visual dialog, reaching SOTA on COMET, VisDial and MNIST Dialog.
Problem
Visual dialog requires reasoning over sequences of image-text pairs and dialog history, but well-aligned large-scale image-text dialog data is scarce, making it hard to train large multimodal models.
Approach
Keep modality-specific pretrained models (ViT and Flan-T5) and train only small cross-modulation transformer blocks whose cross-attention uses an adjacency matrix encoding the spatiotemporal-hierarchical (trip/day/event) structure of the multimodal context to retrieve relevant image and text memories for answer generation.
Key results
- On COMET test, the history-to-utterance variant reaches 90.9 API accuracy, 87.3 coref F1, 93.7 slot F1, 80.6 joint accuracy and 0.298 BLEU, above the multimodal baseline (77.5 joint accuracy, 0.251 BLEU).
- On VisDial v1.0 validation, the model reports MRR 52.59 and R@1 42.90, the best among compared methods, while NDCG, R@5, R@10 and mean rank are close to GST.
- On MNIST Dialog, accuracy is 98.6, below CorefNMN (99.3) but above AMEM (87.53).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use structure-aware multimodal sequential learning | Visual dialog on COMET · COMET test | BLEU: 0.251 -> 0.298 | Helped |
| use structure-aware multimodal sequential learning | Visual dialog on VisDial · VisDial v1.0 validation | MRR: 52.06 -> 52.59 | Helped |
| use structure-aware multimodal sequential learning | Visual dialog on MNIST Dialog · MNIST Dialog test | Accuracy: 99.3 -> 98.6 | Hurt |
| add hierarchical context structure (ablation) | Structure-aware model on COMET · COMET test | BLEU: 0.291 -> 0.298 | Helped |
| use event-level rather than full hierarchy (ablation) | Structure-aware model on COMET · COMET test | BLEU: 0.298 -> 0.296 | No clear effect |
| use day-level rather than full hierarchy (ablation) | Structure-aware model on COMET · COMET test | BLEU: 0.298 -> 0.298 | No clear effect |
| use trip-level rather than full hierarchy (ablation) | Structure-aware model on COMET · COMET test | BLEU: 0.298 -> 0.299 | No clear effect |
| use concatenation cross-modulation structure (ablation) | Structure-aware model on COMET · COMET test | BLEU: 0.298 -> 0.294 | No clear effect |
| reverse cross-modulation order to utterance → history (ablation) | Structure-aware model on COMET · COMET test | BLEU: 0.298 -> 0.296 | No clear effect |
Surprising findings
- Ablation shows removing the hierarchical structure gives the lowest COMET performance, while the trip-level hierarchy performs best among the settings tested.
- On MNIST Dialog the method does not beat the CorefNMN baseline, which uses a memory reference pool and turn-difference information.
Limitations
- Visual dialog results on VisDial are close to the prior GST method on several metrics and the method does not beat CorefNMN on MNIST Dialog.
- Evaluated on only three datasets, with COMET being small (11.5k dialogs).
Experimental setup
Datasets: COMET VisDial v1.0 MNIST Dialog
Benchmarks: COMET VisDial v1.0 MNIST Dialog
Metrics: API call accuracy coreference F1 slot F1 joint accuracy BLEU BERTScore NDCG MRR Recall@k mean rank
Baselines: Text-only (Kottur et al. 2022) Multimodal (Kottur et al. 2022) GST LTMI-GoG DAM CorefNMN AMEM
Builds on: BLIP-2 Flan-T5 ViT Transformer
Closest papers by meaning
- V^2Dial: Unification of Video and Visual Dialog via Multimodal Experts (CVPR 2025 · similarity 0.91)
- F2RVLM: Boosting Fine-grained Fragment Retrieval for Multi-Modal Long-form Dialogue with Vision Language Model (AAAI 2026 · similarity 0.90)
- Cross-modal Information Flow in Multimodal Large Language Models (CVPR 2025 · similarity 0.89)
- MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs (NeurIPS 2024 · similarity 0.88)
- Vision-Speech Models: Teaching Speech Models to Converse about Images (CVPR 2026 · similarity 0.88)
- Towards General Continuous Memory for Vision-Language Models (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.