Research map / Vision-language and multimodal / Multimodal LLMs

OmniResponse: Online Multimodal Conversational Response Generation in Dyadic Interactions

NeurIPS 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces OMCRG, an online task generating synchronized verbal and facial listener responses, and OmniResponse, an MLLM with Chrono-Text Markup and TempoVoice modules, plus the ResponseNet dataset of 696 dyadic videos.

Problem

Generating synchronized verbal (audio) and non-verbal (facial) listener feedback online in dyadic conversations, where audio-face alignment is hard because generation must be incremental and audio is semantically ambiguous; existing datasets lack split-screen, separated-audio, word-level text for bo

Approach

Uses text as an intermediate modality: an MLLM (Phi-3.5 mini) generates listener text and facial coefficients autoregressively, with Chrono-Text Markup ([PAUSE]/[LASTING] tokens) time-aligning text to video frames, and TempoVoice, a decoder that turns text hidden states into Spark-TTS audio tokens synchronized with the face.

Key results

• OmniResponse reaches METEOR 0.141 and BERTScore F1 0.806 on ResponseNet, versus 0.042 / 0.716 for the LSTM and 0.030 / 0.662 for the audio-visual LLM baseline.
• Chrono-Text Markup lowers LSE-D from 11.51 to 9.56 when combined with TempoVoice (ablation: 13.64 without either module), and TempoVoice raises UTMOSv2 from 1.23 to 1.41 in the full model.
• In a 49-participant A/B user study, OmniResponse was preferred over the audio-visual LLM for audio-visual synchronization 95.9% of the time.

Measured effects reported in the paper

Surprising findings

• The LSTM baseline achieves a lower FD (6.51) than OmniResponse (15.46), which the authors attribute to repetitive static visual output.

Limitations

• Performance may depend on training data quality and diversity and on accurate speaker-listener segmentation.
• Degrades in noisy or overlapping conversations and in fast-changing or emotionally rich interactions.
• No fairness analysis is provided.

Experimental setup

Datasets: ResponseNet MultiDialog IEMOCAP ViCo REACT2024 ICD

Benchmarks: ResponseNet test set

Metrics: METEOR BERTScore F1 ROUGE-L Distinct-2 LSE-D UTMOSv2 FD FVD LSE-C

Baselines: LSTM Audio-visual LLM Moshi ReactFace ViCo GPT-4o GPT-4 GPT-o1 Qwen-7B-Chat Claude-Sonnet-4 Gemini-2.5-Flash DeepSeek-R1

Builds on: Phi-3.5 Mini-Instruct Spark-TTS MediaPipe VQGAN TimeMarker

Closest papers by meaning

• GLARE: Generating Listening Heads with Appropriate REactions (NeurIPS 2026 · similarity 0.88)
• OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-time Emotional Speech Synthesis (NeurIPS 2025 · similarity 0.87)
• INFP: Audio-Driven Interactive Head Generation in Dyadic Conversations (CVPR 2025 · similarity 0.87)
• SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words (NeurIPS 2024 · similarity 0.86)
• LLM-driven Multimodal and Multi-Identity Listening Head Generation (CVPR 2025 · similarity 0.86)
• MultiTalk: Scaling Full-Duplex Speech Models to Long, Multi-Party, Bilingual Conversation (NeurIPS 2026 · similarity 0.86)