Research map / Vision-language and multimodal / Multimodal LLMs

EmpathyChat: Structured Cognitive Reasoning in Empathetic Spoken Dialogue

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes EchoChat, a structured perception-to-mental-state-to-response reasoning framework for empathetic spoken dialogue, trained with acoustic-anchored attention SFT and stage-wise RL (cascaded gating plus step-decomposed credit assignment), and releases EchoDialogue-400K and the EchoEval benchmark.

Problem

Current SpeechLLMs treat empathetic spoken dialogue as direct input-to-response mapping, yielding superficially warm responses and cascaded errors across perception, mental-state reasoning, and response stages that trajectory-level training cannot localize.

Approach

Model empathy as a staged cognitive reasoning trace, add an attention regularizer that ties perception tokens to acoustic tokens during SFT, and in RL gate later-stage rewards on earlier-stage quality while giving each reasoning span its own group-relative advantage instead of one broadcast scalar.

Key results

• EchoChat reaches 8.23 perception average and 8.43 reasoning average on EchoEval, versus 7.16 overall for the best baseline Mimo-Audio-7B-Instruct on the All average (8.64 for EchoChat).
• Acoustic-Anchored Attention raises emotion accuracy from 65% to 77% and intensity accuracy from 72% to 91%.
• Cascaded error ratio drops from 0.41 (baseline) to 0.21 for EchoChat on the error propagation analysis.

Limitations

• Judge-based evaluation relies on LLM judges (GPT-4o, Gemini 2.5 Pro) and a human check on a subset only
• Training data is largely synthetic (LLM-generated text and TTS audio)
• Human evaluation covers only 120 samples across multi-turn dialogues

Experimental setup

Datasets: EchoDialogue-400K EchoEval

Benchmarks: EchoEval VStyle

Metrics: perception score (0-10) reasoning score (0-10) response content score (1-5) response audio score (1-5) Spearman correlation with human judgments human MOS-style ratings

Baselines: GLM-4-Voice Step-Audio2 mini Kimi-Audio Mimo-Audio-7B-Instruct Covo-Audio Fun-Audio-Chat OpenS2S MERaLiON BLSP-Emo MiniCPM-O-4.5 Qwen2.5-Omni-7B Qwen3-Omni

Builds on: GRPO Chain-of-Thought Kimi-Audio Qwen2.5-Omni Theory of Mind

Closest papers by meaning

• Nano-EmoX: Unifying Multimodal Emotional Intelligence from Perception to Empathy (CVPR 2026 · similarity 0.88)
• The Illusion of Empathy: How AI Chatbots Shape Conversation Perception (AAAI 2025 · similarity 0.87)
• HumanSense: From Multimodal Perception to Empathetic Context-Aware Responses Through Reasoning MLLMs (AAAI 2026 · similarity 0.87)
• RLVER: Reinforcement Learning with Verifiable Emotion Rewards for Empathetic Agents (ICLR 2026 · similarity 0.86)
• Apathetic or Empathetic? Evaluating LLMs' Emotional Alignments with Humans (NeurIPS 2024 · similarity 0.86)
• MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models (ICLR 2026 · similarity 0.86)