Research map / Vision-language and multimodal / Multimodal LLMs

U-Mind: A Unified Framework for Real-Time Multimodal Interaction with Audiovisual Generation

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Presents U-Mind, a unified real-time system that jointly generates text, speech, motion, and rendered video via a text-first CoT decoding pipeline, trained with rehearsal-driven pretraining and segment-wise alignment on an AnyGPT (LLaMA2-7B) backbone.

Problem

Real-time full-stack multimodal interaction needs high-level reasoning, instruction following, and synchronized speech, gesture, and video output, but existing systems are unimodal or degrade reasoning and cross-modal alignment.

Approach

Builds on AnyGPT-style discrete tokens for text, speech, and motion in one LLaMA2-7B vocabulary; uses rehearsal of pure-text reasoning data during pretraining, segment-wise alignment by prosodic boundaries, and a text-first decoding pipeline that emits a CoT plan before speech and motion tokens, then renders video from pose.

Key results

• On S2M, U-Mind reports FGD 11.12 and angle error 0.188, beating LOM (FGD 16.47, angle error 0.251).
• On multimodal dialogue, U-Mind reaches FGD 7.67 and naturalness 8.11 versus SOLAMI's 18.43 FGD and 5.62 naturalness, though LLM+TTS+LOM has higher relevance (8.72 vs 8.23).
• On T2M, SOLAMI has lower FGD (8.64 vs 12.69) while U-Mind has the best angle error (0.109) and diversity (10.71).

Surprising findings

• Removing text-first decoding drops relevance sharply from 8.23 to 1.24 in the ablation.
• SOLAMI achieves lower FGD than U-Mind on text-to-motion (8.64 vs 12.69).

Limitations

• Motion expressiveness is limited by the discrete motion quantizer vocabulary, so fine facial and hand detail is lost.
• Pretraining data mixture was chosen empirically, with no principled balancing method.
• Evaluation relies on a Qwen LLM judge and dataset-level metrics, with mixed results against baselines on T2M FGD.

Experimental setup

Datasets: BEAT v2 HumanML3D OpenOrca Common Voice Qwen3-generated QA triplets proprietary 400-hour human recordings

Benchmarks: BEAT v2 (S2M) HumanML3D (T2M) multimodal dialogue set instruction-following set

Metrics: FGD Diversity Angle Error Relevance Naturalness

Baselines: SOLAMI LOM EMAGE CaMN DisCo LLM+TTS+LOM

Builds on: AnyGPT SOLAMI LOM SpeechTokenizer DWPose

Closest papers by meaning

• HumanSense: From Multimodal Perception to Empathetic Context-Aware Responses Through Reasoning MLLMs (AAAI 2026 · similarity 0.87)
• UniM: A Unified Any-to-Any Interleaved Multimodal Benchmark (CVPR 2026 · similarity 0.87)
• ROVER: Benchmarking Reciprocal Cross-Modal Reasoning for Omnimodal Generation (ICLR 2026 · similarity 0.87)
• UALM: Unified Audio Language Model for Understanding, Generation and Reasoning (ICLR 2026 · similarity 0.86)
• Video-Only ToM: Enhancing Theory of Mind in Multimodal Large Language Models (CVPR 2026 · similarity 0.86)
• ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoning (ICLR 2026 · similarity 0.86)