Research map / Large language models / Language, safety and interpretability

Large Language Models are Efficient Learners of Noise-Robust Speech Recognition

ICLR 2024 · Large language models › Language, safety and interpretability · approach: Speech and audio language models · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Extends generative error correction for ASR to noisy conditions with a new RobustHP dataset and proposes RobustGER, which derives a language-space noise embedding from N-best hypothesis diversity and distills audio noise information into it via MINE, improving WER with LLaMA-Adapter finetuning.

Problem

Generative error correction (GER) with LLMs over ASR N-best lists performs poorly in noisy conditions, where audio-space noise embeddings can destabilize LLM finetuning due to the cross-modality gap.

Approach

Extract a noise embedding in language space from the diversity of the N-best hypotheses (utterance- and token-level SBERT differences), use it as a subtractive "denoising" prompt in LLaMA-Adapter, and distill real audio noise information into it via mutual information neural estimation (MINE).

Key results

• RobustGER with LLaMA-2-7b reduces average WER on CHiME-4 from 12.8% (baseline) to 5.9%, a 53.9% relative reduction, versus 6.9% for plain GER.
• On NOIZEUS, RobustGER achieves 12.6% average WER vs 13.8% for GER and 14.5% baseline with LLaMA-2-7b.
• Data efficiency: on CHiME-4 with about 5k pairs RobustGER reaches 6.3% average WER vs 5.9% with full data and 6.9% for GER with full data.

Measured effects reported in the paper

Surprising findings

• Directly adding audio-space noise embeddings to LLM finetuning (audio denoising) gives only marginal gains and suffers from the cross-modality gap.
• Utterance-level noise embedding alone yields only minor improvements over vanilla GER, while token-level embedding drives most of the gain.

Limitations

• Gains on some noise conditions (e.g., traffic, babble in LS-FreeSound, car in NOIZEUS) are small or absent.
• Relies on a Whisper beam-search N-best list; the noise embedding depends on the ASR's hypothesis diversity.
• Results are on 7B-13B LLaMA-family and Falcon models; the paper notes instruction template choice has slight impact, left as an open question.

Experimental setup

Datasets: CHiME-4 VoiceBank-DEMAND NOIZEUS LibriSpeech-FreeSound RATS HyPoradise

Benchmarks: CHiME-4 VB-DEMAND NOIZEUS LS-FreeSound RATS

Metrics: word error rate relative WER reduction (GER improvement) N-best oracle WER compositional oracle WER

Baselines: Whisper ASR baseline LM rescoring GER (LLaMA-Adapter finetuning) audio-space denoising ChatGPT in-context learning

Builds on: HyPoradise LLaMA-Adapter Whisper MINE

Closest papers by meaning

• Speech Recognition Meets Large Language Model: Benchmarking, Models, and Exploration (AAAI 2025 · similarity 0.89)
• It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognition (ICLR 2024 · similarity 0.88)
• ASER: Activation Smoothing and Error Reconstruction for Large Language Model Quantization (AAAI 2025 · similarity 0.87)
• An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence (NeurIPS 2026 · similarity 0.87)
• FastLongSpeech: Enhancing Large Speech-Language Models for Efficient Long-Speech Processing (NeurIPS 2025 · similarity 0.87)
• MATS: An Audio Language Model under Text-only Supervision (ICML 2025 · similarity 0.87)