Research map / Large language models / Alignment and preferences
SpeechAlign: Aligning Speech Generation to Human Preferences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SpeechAlign, an iterative self-improvement method that builds a preference codec dataset (golden vs. synthetic first-layer codec tokens) and applies DPO, CoH, RLHF-PPO, or best-of-N to align a SpeechGPT-based codec language model.
Problem
Neural codec language models for speech suffer a distribution gap between golden AR codec tokens (training) and synthetic AR tokens (inference), and speech outputs are rarely aligned to human preferences.
Approach
Treat golden codec tokens from real speech as preferred and model-generated synthetic tokens as dispreferred, avoiding human annotation of codec tokens, then iteratively run preference optimization (mainly DPO) so the model converts weak codec LMs into stronger ones.
Key results
- SpeechAlign-DPO-Iter3 reaches WER 6.0 and SIM 0.90 on LibriSpeech, versus 7.2 and 0.87 for the SFT baseline.
- On VCTK (unseen speakers), SpeechAlign-DPO-Iter3 reaches WER 7.9 and SIM 0.83 versus 8.8 and 0.79 for the SFT baseline.
- On a 130M-parameter AR model, one DPO iteration lowers WER from 11.4 to 9.3 and raises SIM from 0.87 to 0.88.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace baseline SFT with DPO through iteration 3 | SpeechAlign-sft · LibriSpeech test-clean | WER: 7.2 -> 6.0 | Helped |
| replace baseline SFT with DPO through iteration 3 | SpeechAlign-sft · VCTK | WER: 8.8 -> 7.9 | Helped |
| increase DPO iterations from 1 to 3 | SpeechAlign · LibriSpeech test-clean | WER: 6.7 -> 6.0 | Helped |
| increase DPO iterations from 1 to 3 | SpeechAlign · VCTK | WER: 8.5 -> 7.9 | Helped |
| condition the NAR model on golden rather than synthetic AR tokens | SoundStorm NAR model · NAR reconstruction | WER: 7.2 -> 5.9 | Helped |
| condition the NAR model on golden rather than synthetic AR tokens | SoundStorm NAR model · NAR reconstruction | SIM: 0.87 -> 0.93 | Helped |
| increase preference data from 0 to 50k (ablation) | SpeechAlign DPO · zero-shot TTS | WER | Helped |
| increase preference data from 50k to 250k (ablation) | SpeechAlign DPO · zero-shot TTS | WER | No clear effect |
| continue supervised fine-tuning on ground-truth AR tokens instead of preference-aware optimization (ablation) | SpeechAlign-sft · LibriSpeech test-clean | WER: 7.2 -> 8.0 | Hurt |
| continue supervised fine-tuning on ground-truth AR tokens instead of preference-aware optimization (ablation) | SpeechAlign-sft · VCTK | WER: 8.8 -> 9.8 | Hurt |
| apply DPO in the first iteration | 130M-parameter AR model · LibriSpeech test-clean | WER: 11.4 to 9.3 | Helped |
| apply DPO in the first iteration | 130M-parameter AR model · LibriSpeech test-clean | SIM: 0.87 to 0.88 | Helped |
Surprising findings
- Continued SFT on ground-truth tokens does not improve and may degrade speech quality (e.g., LibriSpeech WER 8.0 vs 7.2), so gains come from preference-aware learning.
- Scaling preference data from 50k to 250k in one DPO epoch gives no further gains, suggesting a data-size threshold.
Limitations
- Preference signal is a proxy (golden vs synthetic tokens), not real human preference data; the paper notes fine-grained real human preferences are needed.
- Preference optimization is applied only to the AR model; NAR model distribution gap is left unaddressed.
- Some SpeechAlign variants (e.g., CoH, BoN) show higher WER than the baseline on some benchmarks.
Experimental setup
Datasets: LibriSpeech VCTK Multilingual LibriSpeech
Benchmarks: LibriSpeech test-clean VCTK
Metrics: WER speaker similarity (SIM) human win rate
Baselines: SpeechAlign-sft Continue SFT SpeechAlign-CoH SpeechAlign-RLHF-PPO SpeechAlign-BoN
Builds on: SpeechGPT SoundStorm SpeechTokenizer Direct Preference Optimization Chain-of-Hindsight
Closest papers by meaning
- Aligning Spoken Dialogue Models from User Interactions (ICML 2025 · similarity 0.91)
- Leveraging robust optimization for llm alignment under distribution shifts (NeurIPS 2025 · similarity 0.89)
- Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment (NeurIPS 2024 · similarity 0.89)
- Direct Density Ratio Optimization: A Statistically Consistent Approach to Aligning Large Language Models (ICML 2025 · similarity 0.89)
- Preference Ranking Optimization for Human Alignment (AAAI 2024 · similarity 0.88)
- ComPO: Preference Alignment via Comparison Oracles (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.