Research map / Large language models / Language, safety and interpretability

HALL-E: Hierarchical Neural Codec Language Model for Minute-Long Zero-Shot Text-to-Speech Synthesis

ICLR 2025 · Large language models › Language, safety and interpretability · approach: Speech and audio language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MReQ, a post-training method that restructures a pretrained RVQ neural codec (Encodec/SpeechTokenizer) into a multi-resolution hierarchy reaching 8 Hz, and HALL-E, a VALL-E-style hierarchical LLM-based TTS that enables minute-long zero-shot speech, plus the 40k-hour MinutesSpeech dataset.

Problem

Autoregressive LLM-based TTS models struggle to synthesize long (minute-scale) speech because the high frame rate of neural audio codec tokens makes sequences too long, and naively lowering the codec frame rate degrades quality and pronunciation.

Approach

Post-train a pretrained RVQ codec with a nested multi-resolution RVQ (MRVQ) module, using teacher-student feature distillation (FLD and HSR losses), so the first quantization layer runs at 8 Hz; then build an AR/NAR TTS model (HALL-E) that predicts these hierarchical low-rate tokens via frozen MRVQ sub-modules, initialized from a pretrained VALL-E.

Key results

• MReQ-Encodec reduces the first-layer frame rate to 8 Hz while keeping LibriSpeech reconstruction WER 2.02 and PESQ 3.47, close to the 48 Hz Encodec baseline (WER 2.00, PESQ 3.45).
• On MinutesSpeech test-90s, HALL-E achieves WER 9.79 versus 39.77 for VALL-E (train-28s) and ground truth 10.30, with DNSMOS 3.91.
• HALL-E runs at real-time factor 0.062 versus 0.213 for VALL-E on an RTX 4090, about 3.4x faster.

Measured effects reported in the paper

Surprising findings

• Lowering the frame rate of the AR model to 8 Hz made training stable where training VALL-E on 90s-long audio at high frame rate was unstable (WER 16.14 with train-90s).
• Removing the FLD loss improved WER and SIM on MinutesSpeech test-90s but worsened WD and DNSMOS.

Limitations

• Speaker similarity (SIM) is lower than VALL-E because lower frame rate sacrifices acoustic information.
• WER on LibriSpeech read speech remains above ground truth (under 5% vs 1.74%) due to domain mismatch with spontaneous podcast training data.
• Reducing frame rate to 4 Hz significantly degrades WER (20.07), and reducing bitrate via fewer quantizers hurts WER substantially.

Experimental setup

Datasets: MinutesSpeech LibriSpeech

Benchmarks: MinutesSpeech test-90s MinutesSpeech test-180s LibriSpeech test-clean

Metrics: WER SIM WD DNSMOS QMOS SMOS PESQ RTF

Baselines: VALL-E (Encodec 48Hz) VALL-E (train-28s/54s/90s/180s variants) SpeechTokenizer-based VALL-E

Builds on: VALL-E Encodec SpeechTokenizer MusicGen

Closest papers by meaning

• CLaM-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-Speech (ICLR 2024 · similarity 0.89)
• TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling (NeurIPS 2025 · similarity 0.87)
• DELTA-TTS: Adapting Autoregressive Model into a Diffusion Language Model for Text-to-Speech (NeurIPS 2026 · similarity 0.87)
• ELLA-V: Stable Neural Codec Language Modeling with Alignment-Guided Sequence Reordering (AAAI 2025 · similarity 0.86)
• Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model (AAAI 2025 · similarity 0.86)
• Mega-TTS 2: Boosting Prompting Mechanisms for Zero-Shot Speech Synthesis (ICLR 2024 · similarity 0.86)