Research map / Large language models / Language, safety and interpretability
It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UADF, a training-free step-wise late fusion that combines LLM H2T (GER) token distributions with a Whisper-tiny decoder's distribution, weighting the acoustic branch by LLM token-level uncertainty after temperature calibration.
Problem
GER-based LLM error correction for ASR is text-only and ignores acoustic information, causing data uncertainty; existing early and mid fusion suffer from modality laziness and length/alignment issues.
Approach
Late, token-level fusion in autoregressive decoding: calibrate LLM and ASR logits with temperatures, then dynamically set the ASR weight as sigmoid(LLM entropy) minus beta, so the acoustic model contributes more when the LLM is uncertain.
Key results
- UADF reduces WER on ATIS from 1.61% (GER) to 1.24%, a 23.0% relative reduction, and on WSJ from 2.83% to 2.47% (12.7%).
- On CHiME-4 average, UADF achieves 6.08% WER vs 6.75% for GER (9.9% WERR) and 6.17% for static fusion.
- On LRS3 AVSR, UADF averages 9.9% WER with a 22.7% WERR over the visual-only baseline's reference, surpassing static fusion (10.6%) and comparable to MSRL (10.1%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use UADF instead of GER | LLaMA-7B H2T ASR · ATIS test | WER (%): 1.61 -> 1.24 | Helped |
| Use UADF instead of GER | LLaMA-7B H2T ASR · WSJ test | WER (%): 2.83 -> 2.47 | Helped |
| Use UADF instead of static late fusion | LLaMA-7B H2T ASR · ATIS test | WER (%): 1.36 -> 1.24 | Helped |
| Use UADF instead of static late fusion | LLaMA-7B H2T ASR · WSJ test | WER (%): 2.55 -> 2.47 | Helped |
| Add early-fused Wav2vec2-large speech tokens | GER on LLaMA-7B · ATIS and WSJ | WER (%): ATIS: 1.61 -> 2.16; WSJ: 2.83 -> 3.21 | Hurt |
| Add early-fused HuBERT speech tokens | GER on LLaMA-7B · ATIS and WSJ | WER (%): ATIS: 1.61 -> 2.02; WSJ: 2.83 -> 3.11 | Hurt |
| Add Whisper encoder cross-attention | GER on LLaMA-7B · ATIS and WSJ | WER (%): ATIS: 1.61 -> 1.75; WSJ: 2.83 -> 2.59 | Mixed |
| Improve the ASR-only model used for late fusion (ablation) | UADF on ATIS · ATIS test | WER (%): ASR-only WER 12.16, UADF 1.52; ASR-only WER 8.22, UADF 1.39; ASR-only WER 4.67, UADF 1.24 | Helped |
| Add calibrated fixed-weight fusion without dynamic weighting (ablation) | Late fusion on ATIS · ATIS test | WER (%): 1.57 -> 1.33 | Helped |
| Add calibration to dynamic fusion (ablation) | Late fusion on ATIS · ATIS test | WER (%): 1.39 -> 1.24 | Helped |
| Use UADF instead of static fusion | ASR on CHiME-4 · CHiME-4 test-real, average | WER (%): 6.17 -> 6.08 | Helped |
| Use UADF instead of static fusion | AV-HuBERT and V-HuBERT fusion · LRS-3, Babble average | WER (%): 10.6 -> 9.9 | Helped |
Surprising findings
- Early fusion with speech tokens performs worse than the GER text-only baseline, attributed to modality laziness.
- Calibration alone without dynamic weighting (ID-C) does not improve GER over text-only decoding.
Limitations
- Early fusion without the N-best list was not successfully learned by LoRA tuning, left to future work.
- Reliance on a small validation set (200 examples) for temperature and weight calibration and on a per-dataset beta setting.
- Results on AVSR are reported with per-SNR beta tuning, and the gains over MSRL are marginal.
Experimental setup
Datasets: HyPoradise WSJ ATIS CHiME-4 LRS3
Benchmarks: ATIS WSJ CHiME-4 LRS3
Metrics: word error rate word error rate reduction
Baselines: GER early fusion (Wav2vec2/HuBERT tokens) mid fusion (Whispering-LLaMA cross-attention) static late fusion ID-C AV-HuBERT V-HuBERT MSRL LM_rank
Builds on: HyPoradise generative error correction Whispering-LLaMA LoRA multimodal sentiment dynamic fusion (uncertainty-based)
Closest papers by meaning
- Large Language Models are Efficient Learners of Noise-Robust Speech Recognition (ICLR 2024 · similarity 0.88)
- Speech Recognition Meets Large Language Model: Benchmarking, Models, and Exploration (AAAI 2025 · similarity 0.87)
- FastLongSpeech: Enhancing Large Speech-Language Models for Efficient Long-Speech Processing (NeurIPS 2025 · similarity 0.85)
- FUSE: Quantifying Uncertainty in Vision-Language Models by Bayesian Fusing Epistemic and Aleatoric Uncertainty (ICML 2026 · similarity 0.84)
- Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs (NeurIPS 2024 · similarity 0.84)
- Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation Models (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.