Research map / Large language models / Language, safety and interpretability
Speech Recognition Meets Large Language Model: Benchmarking, Models, and Exploration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically benchmarks LLM-based ASR designs (speech encoders, chat vs pre-trained LLMs, MLP projector) and builds a simple recipe that beats prior NN-based and LLM-based ASR on LibriSpeech and GigaSpeech-M, plus reports capability emergence during training.
Problem
Many design decisions in LLM-based automatic speech recognition are inadequately justified, making it unclear which choices actually improve performance.
Approach
A deliberately minimal setup (off-the-shelf speech encoder, temporal downsampling, single trainable MLP projector, frozen LLM) with systematic ablations over encoders and LLMs shows that chat/SFT LLMs and fine-tuned self-supervised encoders work best, without task-specific tricks.
Key results
- Best system (WavLM Large encoder, Vicuna-7B, MLP projector) reaches 1.96% / 4.18% WER on LibriSpeech test-clean/test-other; with HuBERT X-Large encoder 1.8% / 3.4%, beating Zipformer-large with in-domain LM (1.9/3.9).
- On GigaSpeech, the LLM-based model reaches 10.6 DEV / 11.1 TEST WER, better than Kaldi, Fairseq Conformer-Transducer and K2 Zipformer baselines but behind Whisper-large-v2 (10.5/10.2).
- Chat LLMs outperform raw pre-trained LLMs for ASR (e.g., LLaMA-2-Chat 2.72/6.79 vs LLaMA-2 3.01/7.15 with Whisper-large), and Vicuna gives the best 2.58/6.47.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use an SFT chat LLM instead of a raw pre-trained LLM | LLM-based ASR with Whisper encoders · Librispeech test-clean and test-other | WER (%): 3.01 -> 2.72; 7.15 -> 6.79 | Helped |
| Use Vicuna instead of LLaMA-2 | LLM-based ASR with Whisper-large-v2 · Librispeech test-clean and test-other | WER (%): 3.01 -> 2.58; 7.15 -> 6.47 | Helped |
| Fine-tune HuBERT Large on Librispeech 960 hours | LLM-based ASR with Vicuna-7B · Librispeech test-clean and test-other | WER (%): 4.53 -> 2.10; 8.74 -> 4.26 | Helped |
| Fine-tune WavLM Large on Librispeech 960 hours | LLM-based ASR with Vicuna-7B · Librispeech test-clean and test-other | WER (%): 2.13 -> 1.96; 4.73 -> 4.18 | Helped |
| Add an LLM rather than use CTC without an LM (ablation) | WavLM Large ASR · Librispeech and GigaSpeech | WER (%): 2.66 -> 1.96; 4.97 -> 4.18; 12.77 -> 10.63; 12.77 -> 11.05 | Helped |
| Use an LLM instead of an in-domain LM (ablation) | WavLM Large ASR · Librispeech and GigaSpeech | WER (%): 2.14 -> 1.96; 4.00 -> 4.18; 11.01 -> 10.63; 11.45 -> 11.05 | Mixed |
| Use LLM-based ASR instead of Zipformer-Pruned RNN-T | GigaSpeech ASR · GigaSpeech TEST | WER (%): 12.19 -> 11.1 | Helped |
| Use LLM-based ASR instead of Whisper-large-v2 | GigaSpeech ASR · GigaSpeech TEST | WER (%): 10.2 -> 11.1 | Hurt |
Surprising findings
- Chat LLMs beat pre-trained LLMs on ASR despite the pre-trained LLM having lower transcript perplexity, ruling out text leakage as the explanation.
- Dropping the speech encoder and feeding raw FBank features to the projector performs far worse (68.95/99.37 WER).
Limitations
- LLM-based ASR still underperforms Whisper-large-v2 on noisy GigaSpeech data.
- Inference speed is a bottleneck, as acknowledged by the authors.
- Only MLP projector is studied in the main benchmark; alternatives like Q-Former are dismissed as less stable.
Experimental setup
Datasets: LibriSpeech-960 GigaSpeech-M Libri-Light (pre-training of HuBERT) LibriSpeech LM corpus
Benchmarks: LibriSpeech test-clean LibriSpeech test-other GigaSpeech DEV GigaSpeech TEST
Metrics: word error rate word-level perplexity training next-token accuracy
Baselines: ContextNet Conformer Branchformer Zipformer Whisper-large-v2 OWSM-v3.1 Yu et al. 2024 SALMONN Qwen-Audio
Builds on: Whisper WavLM HuBERT Vicuna LLaMA-2
Closest papers by meaning
- Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs (NeurIPS 2024 · similarity 0.90)
- FastLongSpeech: Enhancing Large Speech-Language Models for Efficient Long-Speech Processing (NeurIPS 2025 · similarity 0.89)
- Large Language Models are Efficient Learners of Noise-Robust Speech Recognition (ICLR 2024 · similarity 0.89)
- Zipformer: A faster and better encoder for automatic speech recognition (ICLR 2024 · similarity 0.89)
- Is Text All You Need? Text as a Universal Information Bottleneck for Speech LLMs (NeurIPS 2026 · similarity 0.88)
- Speech Robust Bench: A Robustness Benchmark For Speech Recognition (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.