Research map / Vision-language and multimodal / Multimodal LLMs

SVHalluc: Benchmarking Speech-Vision Hallucination in Audio-Visual Large Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces SVHalluc, a 2405-question benchmark on YouCook2 that tests whether audio-visual LLMs ground speech content in video across semantic and temporal hallucination tasks, finding open-source models near random and Gemini 2.5 Pro far better.

Problem

Audio-visual LLMs may hallucinate when speech content is involved, but existing hallucination benchmarks only use environmental sounds as event indicators, leaving speech-vision alignment (semantic and temporal) unevaluated.

Approach

Builds speech-vision hallucination tests from YouCook2 by pairing speech with mismatched or time-shifted video and by extracting visible vs. speech-only actions and objects, organized into six coarse-to-fine tasks across semantic and temporal dimensions.

Key results

• Gemini 2.5 Pro reaches 93.10% accuracy on global semantic alignment, while open-source models such as video-SALMONN and Qwen3-Omni score around 50-55% on that task.
• On temporal alignment, Gemini 2.5 Pro achieves 85.17% accuracy versus 50-51% for most open-source models.
• Qwen3-Omni reaches 0.8844 mIoU with Silero-VAD for speech timestamps, suggesting speech localization is not the main bottleneck.

Surprising findings

• Adding speech transcripts to Qwen3-Omni lowers accuracy on the cross-modal temporal binding task, since speech acts as a distractor.
• Video-only input improves Qwen3-Omni on FGSA and CMSB over the full audio-visual input, indicating speech can hurt performance on these tasks.

Limitations

• Built only from YouCook2 cooking videos, limiting domain diversity.
• Relies on GPT for object/action extraction and filtering, with human verification only as final check.
• Open-source models evaluated are a limited set; results may vary with prompts and versions.

Experimental setup

Datasets: YouCook2

Benchmarks: SVHalluc

Metrics: accuracy precision recall F1 word error rate word information preserved mIoU

Baselines: Gemini 2.5 Pro video-SALMONN video-SALMONN 2 VideoLLaMA 2 Qwen2.5-Omni

Builds on: YouCook2 AVHBench POPE

Closest papers by meaning

• AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Models (ICLR 2025 · similarity 0.94)
• Visual Hallucination Elevates Speech Recognition (AAAI 2024 · similarity 0.93)
• VidHalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understanding (CVPR 2025 · similarity 0.92)
• AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decoding (NeurIPS 2025 · similarity 0.91)
• Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.91)
• Analyzing and Mitigating Object Hallucination: A Training Bias Perspective (AAAI 2026 · similarity 0.91)