Research map / Vision-language and multimodal / CLIP and zero-shot
ELLA-V: Stable Neural Codec Language Modeling with Alignment-Guided Sequence Reordering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ELLA-V, a zero-shot LM-based TTS that interleaves phoneme and EnCodec acoustic tokens (phonemes placed before their audio) and computes loss only on acoustic and EOP/EOS tokens, improving robustness over VALL-E.
Problem
VALL-E-style zero-shot TTS suffers from alignment errors (repetitions, transpositions, omissions), weak fine-grained phoneme-level control, and infinite silence generation, especially under greedy decoding.
Approach
Reorders the input sequence so each phoneme token sits directly before its aligned acoustic tokens (sandwiched by EOP), prepends a global phoneme sequence ("global advance"), optionally shifts phoneme/EOP tokens forward ("local advance"), and trains the GAR model only on acoustic, EOP and EOS targets so the model can truncate abnormal phonemes.
Key results
- On the LibriSpeech test-clean continuation task, ELLA-V reduces WER to 2.28% versus 5.00% for the reproduced VALL-E, with comparable speaker similarity (0.870 vs 0.868) and CMOS +0.10.
- On 100 hard synthesis cases, ELLA-V achieves 12.79% WER versus 28.39% for VALL-E.
- Under decreasing nucleus top-p (down to greedy), VALL-E's WER rises to 49.26% with 87.29% infinite-silence rate in the motivating table, while ELLA-V remains stable across decoding strategies.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace VALL-E with ELLA-V | Zero-shot TTS · LibriSpeech test-clean continuation | WER (%): 5.00 -> 2.28 | Helped |
| replace VALL-E with ELLA-V | Zero-shot TTS · 100 hard synthesis cases | WER (%): 28.39 -> 12.79 | Helped |
| replace VALL-E with ELLA-V | Zero-shot TTS · LibriSpeech test-clean continuation | SPK: 0.868 -> 0.870 | No clear effect |
| replace VALL-E with ELLA-V | Zero-shot TTS · LibriSpeech test-clean continuation | SMOS: 3.56 -> 3.56 | No clear effect |
| replace VALL-E with ELLA-V | Zero-shot TTS · LibriSpeech test-clean continuation | CMOS: 0.00 -> 0.10 | Helped |
| use lower top-p sampling thresholds (ablation) | VALL-E · VALL-E zero-shot TTS decoding | WER (%): 5.47 -> 49.26 | Hurt |
| use lower top-p sampling thresholds (ablation) | VALL-E · VALL-E zero-shot TTS decoding | INF (%): 0.00 -> 87.29 | Hurt |
| add the global phoneme sequence at the beginning (ablation) | ELLA-V · LibriSpeech test-clean continuation | WER (%): 5.00 -> 2.28 | Helped |
| add the global phoneme sequence at the beginning (ablation) | ELLA-V · LibriSpeech test-clean continuation | SPK: 0.859 -> 0.870 | Helped |
| add phoneme tokens after BOS before their corresponding acoustic tokens (ablation) | ELLA-V · LibriSpeech test-clean continuation | WER (%): 3.51 -> 2.28 | Helped |
| add phoneme tokens after BOS before their corresponding acoustic tokens (ablation) | ELLA-V · LibriSpeech test-clean continuation | SPK: 0.868 -> 0.870 | Helped |
| use local advance of 5 or 10 tokens (ablation) | ELLA-V · Zero-shot TTS decoding across top-p values | robustness | Helped |
Surprising findings
- Removing the global phoneme prefix (ELLA-V-noglobal) raises WER to 5.00%, equal to VALL-E, showing the global phoneme information is key.
- VALL-E's greedy decoding yields 87.29% infinite-silence samples, far worse than sampling-based decoding.
Limitations
- Relies on Montreal Forced Aligner for alignment and sentences with unrecognized phones are excluded.
- Evaluation uses a self-reproduced VALL-E trained on LibriSpeech 960h rather than the original 60k-hour setup, and the hard-case set is self-constructed.
- Subjective CMOS gain over VALL-E is small (+0.10) and SMOS is tied.
Experimental setup
Datasets: LibriSpeech 960h LibriLight (VALL-E original training data, cited) LibriSpeech test-clean
Benchmarks: LibriSpeech test-clean (continuation task) 100 hard synthesis cases
Metrics: WER SPK similarity (WavLM-TDNN) INF% CUT% CMOS SMOS
Baselines: VALL-E Ground Truth-Encodec
Builds on: VALL-E EnCodec Montreal Forced Aligner UniLM
Closest papers by meaning
- CLaM-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-Speech (ICLR 2024 · similarity 0.91)
- Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model (AAAI 2025 · similarity 0.87)
- MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer (ICLR 2025 · similarity 0.87)
- HALL-E: Hierarchical Neural Codec Language Model for Minute-Long Zero-Shot Text-to-Speech Synthesis (ICLR 2025 · similarity 0.86)
- ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling (ICML 2025 · similarity 0.86)
- A Unified Audio Language Model with Text-Aligned Factorized Audio Tokenization (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.