Research map / Large language models / Language, safety and interpretability

MultiTalk: Scaling Full-Duplex Speech Models to Long, Multi-Party, Bilingual Conversation

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Speech and audio language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds an LLM-scripted, IndexTTS2-rendered synthetic data engine producing a 57.6k-hour bilingual multi-party full-duplex corpus, a long multi-party bilingual benchmark (MultiTalkBench), and a Moshi-based model trained in two phases that beats open baselines on that benchmark.

Problem

Open-source full-duplex speech models lack long-context, multi-party, and bilingual capability, and there is little open training data or evaluation for this setting.

Approach

Synthesize parallel-stream multi-party bilingual dialogue scripts with LLMs, render them with multi-speaker TTS and multi-channel mixing with word alignment, then train a Moshi model in two phases (bilingual dyadic pretraining, multi-party fine-tuning) and evaluate on real-recording-based long multi-party benchmark with LLM judging.

Key results

• The proposed model reaches a MultiTalkBench final score of 13.15 versus 4.66 for its Moshiko-7B initialization, 11.05 for MiniCPM-o-4.5, and 4.94 for Qwen3-Omni-30B.
• Chinese final score rises from 0.36 (Moshiko-7B) to 16.27, while English final score rises from 7.82 to 10.87.
• Human reference scores 68.74, showing large headroom; Gemma-4-31B judge reaches Spearman rho 0.898 against humans on the Final Score.

Surprising findings

• Qwen3-Omni-30B scores lower than the 7B Moshi-based model on the benchmark, suggesting multi-party competence is bounded by training distribution rather than parameter count.
• PersonaPlex-7B has the highest participation score (97.87) yet the lowest final score (2.52) among compared models.

Limitations

• Large gap to human reference (13.15 vs 68.74) indicates limited capability on the benchmark.
• Training data is synthetic TTS-rendered while evaluation uses real recordings, a cross-domain setting.
• Evaluated only on a 7B backbone and only English and Chinese; no intra-sentential code-switching tested.

Experimental setup

Datasets: MultiTalkPT MultiTalkFT AMI ICSI CHiME-6 AISHELL-4 AliMeeting AISHELL-5 MagicData-RAMC dair-ai/emotion AM-DeepSeek-R1-0528-Distilled

Benchmarks: MultiTalkBench

Metrics: final score general score multi-party IQ score role-conditional score participation score Spearman correlation

Baselines: Moshiko-7B PersonaPlex-7B MiniCPM-o-4.5 Qwen3-Omni-30B-A3B-Instruct F-Actor

Builds on: Moshi IndexTTS2 Montreal Forced Aligner

Closest papers by meaning

• Language Model Can Listen While Speaking (AAAI 2025 · similarity 0.89)
• MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark (ICLR 2026 · similarity 0.88)
• SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words (NeurIPS 2024 · similarity 0.88)
• SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversation (NeurIPS 2025 · similarity 0.87)
• Long-Form Speech Generation with Spoken Language Models (ICML 2025 · similarity 0.87)
• FastLongSpeech: Enhancing Large Speech-Language Models for Efficient Long-Speech Processing (NeurIPS 2025 · similarity 0.87)