Research map / Large language models / Language, safety and interpretability

Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics

ICLR 2025 · Large language models › Language, safety and interpretability · approach: Speech and audio language models · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a turn-taking evaluation protocol for audio foundation models using a Whisper-based causal turn-taking predictor (trained on Switchboard) as a judge, and uses it to benchmark Moshi, a cascaded VAD-based system, and several audio FMs on understanding and predicting turn-taking events.

Problem

Audio foundation models for spoken dialogue are not evaluated on their turn-taking abilities (when to speak, backchannel, interrupt, handle interruptions), and corpus-level statistics cannot assess the timing of turn-taking events.

Approach

A causal supervised turn-taking predictor (Whisper-medium encoder over 40 ms chunks, trained on Switchboard with mixed single-channel audio) serves as a judge that labels AI turn-taking decisions in human-AI conversations, yielding timing-centric metrics for speaking up, backchanneling, interrupting, conveying floor, and handling interruptions.

Key results

• Supervised judge reaches 92.0 overall ROC-AUC on in-domain Switchboard and 91.5 and 91.0 on Columbia Games and Fisher OOD test sets.
• Moshi interrupts aggressively (2.09% false-positive interruption decisions), and only 17.4% of user interruptions led to turn change; the cascaded system yielded its turn over 50% of the time when interrupted.
• On the understanding task, Whisper+GPT-4o reaches 66.3% on turn change while open-source audio FMs stay near 50% on backchannel and interruption; on prediction, all open-source audio FMs perform close to random.

Measured effects reported in the paper

Surprising findings

• Open-source audio FMs perform close to random chance on predicting turn-taking events, while a cascade of Whisper and GPT-4o does better only on turn change.
• Moshi fails to speak up even when the user is ready to yield, and rarely backchannels.

Limitations

• Judge model agreement with human judgments is below 80% for many metrics, and interruption-handling agreement is only about 57-61%.
• User study relies on a small number of participants, mostly authors and colleagues (11 speakers), limiting generality.
• Backchannel identification relies on heuristic filler-word lists and may miss backchannels; protocol requires supervised data to train the judge.

Experimental setup

Datasets: Switchboard Columbia Games Corpus Fisher Corpus

Benchmarks: Switchboard test set Columbia Games Corpus Fisher Corpus Talking Turns turn-taking understanding/prediction benchmarks

Metrics: ROC-AUC accuracy F1 percentage of turn-taking decisions agreement with judge labels speaking rate backchannel rate

Baselines: Moshi Cascaded VAD-based system SALMONN Qwen2-Audio-Instruct Qwen-Audio-Chat Whisper+GPT-4o Random Baseline

Builds on: Voice Activity Projection Whisper VAP evaluation metrics

Closest papers by meaning

• Language Model Can Listen While Speaking (AAAI 2025 · similarity 0.88)
• Aligning Spoken Dialogue Models from User Interactions (ICML 2025 · similarity 0.88)
• $\tau$-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains (ICML 2026 · similarity 0.87)
• An Efficient Self-Learning Framework For Interactive Spoken Dialog Systems (ICML 2024 · similarity 0.87)
• Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap (NeurIPS 2026 · similarity 0.87)
• Decoupling Conversational Dynamics in Full-Duplex Spoken Models through Reinforcement Learning (NeurIPS 2026 · similarity 0.87)