Research map / Vision-language and multimodal / Human and embodied modeling
Towards Holistic, Pragmatic and Multimodal Conversational Systems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
An AAAI 2024 invited-talk abstract summarizing the author's research on multimodal discourse models: visual context for machine translation, visual refinement of generated text, abstraction-based visio-linguistic systems, and an EEG/ERP study of audio-only vs audio-visual language comprehension.
Problem
Language understanding and generation in conversational systems is limited when it ignores non-lexical modalities such as vision, prosody, and gestures, and it is unclear how multimodal context changes both model design and human language processing.
Approach
Synthesizes prior work by the author: multimodal MT benefits most when modalities are complementary, visual concept abstractions (object categories) improve grounding and interpretability, and surprisal from n-gram versus Transformer LMs predicts N400 responses differently in audio-only versus audio-visual settings.
Key results
- Multimodal MT models perform better when visual and textual modalities are complementary rather than redundant (Caglayan et al. 2019).
- In an audio-only vs audio-visual N400 study, Transformer LMs fit best audio-only, while 2-gram models fit best in the multimodal setting.
- Abstractions such as object categories are argued to make vision-language systems more interpretable and less prone to hallucination (Whitehouse et al. 2023).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use complementary rather than redundant visual context | machine translation models with visual context · machine translation | translation effectiveness | Helped |
| use visual information to refine generated text | multimodal text generation · generated text | alignment with visual concepts | Helped |
| abstract visual information using concepts such as object categories | vision-and-language systems · image captioning and visual question answering | interpretability | Helped |
| abstract visual information using concepts such as object categories | vision-and-language systems · image captioning and visual question answering | hallucinations | Helped |
| use Transformer-based language models rather than 2-gram models | language models predicting N400 responses · audio-only | N400 response fit | Helped |
| use 2-gram language models rather than Transformer-based models | language models predicting N400 responses · multimodal | N400 response fit | Helped |
Surprising findings
- Visual stimuli consistently override auditory stimuli when auditory information is purely complementary, analogous to the dominance effect in psychophysics.
- Local 2-gram language models predict N400 better than Transformers in the multimodal setting, despite Transformers winning in audio-only.
Limitations
- Abstract-only talk summary; no experimental details or numbers given.
- Cited works are referenced rather than described in full.
Closest papers by meaning
- Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark (NeurIPS 2025 · similarity 0.87)
- The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion (CVPR 2025 · similarity 0.86)
- SayNext-Bench: Why Do LLMs Struggle with Next-Utterance Anticipation? (NeurIPS 2026 · similarity 0.85)
- Structure-Aware Multimodal Sequential Learning for Visual Dialog (AAAI 2024 · similarity 0.85)
- Debiased Multimodal Understanding for Human Language Sequences (AAAI 2025 · similarity 0.85)
- OmniResponse: Online Multimodal Conversational Response Generation in Dyadic Interactions (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.