Research map / Large language models / Language, safety and interpretability

SSDM: Scalable Speech Dysfluency Modeling

NeurIPS 2024 · Large language models › Language, safety and interpretability · approach: Speech and audio language models · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SSDM, which uses neural variational articulatory gestural scores as a scalable forced aligner, a Connectionist Subsequence Aligner for dysfluency-aware monotonic alignment, and an LLM (LLaMA-7B with LoRA) for end-to-end dysfluency detection, plus the Libri-Dys simulated corpus.

Problem

Speech dysfluency modeling (detecting dysfluency type and timing) scales poorly with data, lacks large-scale dysfluency corpora, and lacks an effective learning framework.

Approach

Replaces acoustic/SSL features with neural variational gestural scores (articulatory representations obtained via acoustic-to-articulatory inversion, self-distilled from acoustic and text priors) and aligns them to reference phonemes with a CTC-style LCS-constrained aligner (CSA), feeding the alignment to an LLM; also simulates dysfluent speech in text space via StyleTTS2 to create Libri-Dys.

Key results

• On Libri-Dys (100%), SSDM with curriculum reaches F1 81.6% and MS 71.0%, versus H-UDM at 84.1% F1 / 65.2% MS on VCTK++ and 75.9% F1 / 61.1% MS on Libri-Dys in the H-UDM Libri-Dys setting.
• On nfvPPA real speech, SSDM w/ Curri reaches 69.9% F1 and 55.0% MS, while ChatGPT gets 5.6% F1 and LTU-AS-13B and SALMONN-13B get 0% MS.
• Libri-Dys is reported as 3983 hours, about 7x LibriTTS, with a human MOS of 4.15 versus 2.14 for VCTK++.

Measured effects reported in the paper

Surprising findings

• The LLaMA component contributes only marginally; removing LLaMA gives similar detection results, and the authors attribute this to token granularity.
• Off-the-shelf large speech models (LTU-AS-13B, SALMONN-13B) achieve near-zero matching scores on dysfluency detection without fine-tuning.

Limitations

• LLM contribution is marginal; authors suspect token granularity and suggest phoneme-level language models.
• Current data scale is inadequate and constrained by compute.
• Real clinical evaluation is small (about 1 hour from 38 nfvPPA participants), and a large sim-to-real gap remains for real disordered speech.

Experimental setup

Datasets: Libri-Dys LibriTTS VCTK++ nfvPPA

Benchmarks: Libri-Dys VCTK++ nfvPPA

Metrics: F1 dPER Matching Score (MS) scaling factors SF1/SF2 MOS PER

Baselines: HuBERT H-UDM LTU-AS-13B SALMONN-13B ChatGPT DTW (soft-DTW)

Builds on: H-UDM 2D-Alignment CTC LTU-AS StyleTTS2

Closest papers by meaning

• Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks (ICLR 2025 · similarity 0.84)
• Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversation (NeurIPS 2024 · similarity 0.84)
• Speech Recognition Meets Large Language Model: Benchmarking, Models, and Exploration (AAAI 2025 · similarity 0.84)
• Is Text All You Need? Text as a Universal Information Bottleneck for Speech LLMs (NeurIPS 2026 · similarity 0.84)
• DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factors (ICLR 2025 · similarity 0.83)
• Long-Form Speech Generation with Spoken Language Models (ICML 2025 · similarity 0.83)