Research map / Vision-language and multimodal / Cross-modal learning
Visual Hallucination Elevates Speech Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DFVGM, which discretizes audio and visual features with K-means and translates noisy audio tokens into visual "hallucination" tokens so AVSR can run audio-only at inference.
Problem
Audio-visual speech recognition degrades or fails when the paired video is missing at inference, and noisy audio hampers audio-only ASR.
Approach
Model audio-visual semantic correspondences in discrete (K-means codebook) feature spaces rather than continuous ones, generating visual tokens from noisy audio with a transformer with a local-distance cross-attention penalty and a KL consistency loss between ground-truth and pseudo visual token paths.
Key results
- Audio-only with visual hallucinations reduces average WER on LRS2 from 30.5% (Conformer) to 13.9%, about 53% relative to the best AO baseline described in the abstract (reported as 12.9%).
- On LRS3 at -5 dB and -10 dB babble noise, WER drops from 62.3% to 14.3% and 97.5% to 40.6% versus AV-Hubert.
- With ground-truth visual input, the model reaches 9.4% avg WER on LRS2 versus 24.7% for LUSL, outperforming AV baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use visual hallucinations generated from audio instead of audio alone | Ours · LRS2, average across babble-noise SNRs | WER (%): 30.5 -> 13.9 | Helped |
| Use visual hallucinations generated from audio instead of audio alone | Ours · LRS3, average across babble-noise SNRs | WER (%): 36.7 -> 13.4 | Helped |
| Use ground-truth visual sequences rather than generated visual hallucinations | Ours · LRS2, average across SNRs | WER (%): 13.9 -> 9.4 | Helped |
| Use ground-truth visual sequences rather than generated visual hallucinations | Ours · LRS3, average across SNRs | WER (%): 13.4 -> 10.2 | Helped |
| Add 20% of the LRS3 audio dataset for training (ablation) | Ours pretrained on LRS2 · LRS2, 0 dB | WER (%): 12.6 -> 11.2 | Helped |
| Add 40% of the LRS3 audio dataset for training (ablation) | Ours pretrained on LRS2 · LRS2, 0 dB | WER (%): 12.6 -> 10.9 | Helped |
| Add babble-noise augmentation during DFVGM training (ablation) | DFVGM · LRS2 test, babble noise, clean audio | BLEU: 72.3 -> 74.2 | Helped |
| Add babble-noise augmentation during DFVGM training (ablation) | DFVGM · LRS2 test, babble noise, -5 dB | BLEU: 48.1 -> 53.1 | Helped |
| Use distance-penalty cross-attention with R=2 instead of vanilla cross-attention (ablation) | DFVGM · LRS3 | BLEU: +5 BLEU | Helped |
Surprising findings
- Modality dropout (AV-Hubert) falls short of visual-hallucination approach in noisy audio-only settings, despite addressing input discrepancy.
Limitations
- Performance degrades sharply at very low SNR (-10 dB) with WER still around 40%
- Relies on pretrained encoders and an additional language model, and training uses only babble noise for augmentation
Experimental setup
Datasets: LRS2 LRS3 NOISEX babble noise
Benchmarks: LRS2 LRS3
Metrics: WER BLEU
Baselines: Conformer (Ma et al. 2021) AVEC Pseudo Visual (Hegde et al. 2021) AV-Hubert LUSL AV-RelScore
Builds on: AV-HuBERT Conformer hybrid CTC/attention Wav2Vec 2.0 MoCo v2 LUSL
Closest papers by meaning
- SVHalluc: Benchmarking Speech-Vision Hallucination in Audio-Visual Large Language Models (CVPR 2026 · similarity 0.93)
- AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decoding (NeurIPS 2025 · similarity 0.90)
- AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Models (ICLR 2025 · similarity 0.90)
- Multi-Modal Hallucination Control by Visual Information Grounding (CVPR 2024 · similarity 0.88)
- Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.88)
- Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.