Research map / Large language models / Language, safety and interpretability

GenSE: Generative Speech Enhancement via Language Models using Hierarchical Modeling

ICLR 2025 · Large language models › Language, safety and interpretability · approach: Speech and audio language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GenSE, an LM-based speech enhancement framework that maps noisy semantic tokens to clean semantic tokens and then to clean acoustic tokens from a single-quantizer codec (SimCodec), with token chain prompting for speaker consistency.

Problem

Existing speech enhancement methods ignore semantic information and generalize poorly to unseen noise; applying LM-based generation to noisy speech is unstable, and multi-quantizer codecs produce long token sequences that are hard for LMs.

Approach

Decouple denoising from generation via a two-stage hierarchy: an autoregressive noisy-to-semantic LM produces clean XLSR semantic tokens, then a semantic-to-speech LM generates clean acoustic tokens from a single-quantizer SimCodec (8192 codebook via codebook reorganization), prompted by both noisy and clean semantic tokens plus noisy acoustic tokens.

Key results

• On the DNS test set without reverb, GenSE reaches DNSMOS OVL 3.43 and SECS 0.67, outperforming all baselines including SELM (3.21 OVL, 0.62 SECS).
• On the out-of-domain CHiME-4 set, GenSE attains the lowest WER (28.4) and highest SECS (0.62) among compared systems.
• SimCodec at 1.3 kbps (100 token/s, single quantizer) reaches PESQ 3.05 and UTMOS 3.37, beating DAC at 1.0 kbps (PESQ 1.32).

Measured effects reported in the paper

Surprising findings

• Removing hierarchical modeling lowers SECS sharply (0.66 to 0.41 in the ablation), and removing token chain prompting drops SECS to 0.43, which is larger than the DNSMOS change.
• Scaling the single-quantizer codebook to 2^13 reduced codebook usage to about 50%, yet the reorganization strategy recovered usage and quality.

Limitations

• Autoregressive token generation makes inference slow and limits real-time use.
• Limited training data coverage and model size; authors plan to scale up.
• Lower-bandwidth SimCodec variant degrades DNSMOS metrics relative to the full version.

Experimental setup

Datasets: LibriSpeech LibriLight LibriTTS VoiceBank DNS Challenge WHAM! DEMAND openSLR26 openSLR28 CHiME-4

Benchmarks: DNS Challenge test set CHiME-4

Metrics: DNSMOS SECS VQScore WER PESQ STOI MCD UTMOS NMOS SMOS

Baselines: FullSubNet Inter-Subnet CDiffuSE SGMSE StoRM SELM DOSE DAC EnCodec HiFi-Codec SpeechTokenizer Vocos

Builds on: SELM VALL-E AudioLM EnCodec XLSR

Closest papers by meaning

• A Variational Framework for Improving Naturalness in Generative Spoken Language Models (ICML 2025 · similarity 0.87)
• TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling (NeurIPS 2025 · similarity 0.86)
• Metis: A Foundation Speech Generation Model with Masked Generative Pre-training (NeurIPS 2025 · similarity 0.86)
• Large Language Models are Efficient Learners of Noise-Robust Speech Recognition (ICLR 2024 · similarity 0.85)
• Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction (ICML 2025 · similarity 0.85)
• FINALLY: fast and universal speech enhancement with studio-like quality (NeurIPS 2024 · similarity 0.85)