Research map / Vision-language and multimodal / Multimodal LLMs
AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AVHBench, a benchmark of four audio-visual hallucination/matching/captioning tasks built with a semi-automatic GPT-4 and human-verified pipeline, evaluates six audio-visual LLMs, and shows LoRA fine-tuning plus audio alignment reduces cross-modal hallucinations.
Problem
Audio-visual LLMs hallucinate across modalities (hearing imagined sounds from visual cues and seeing imagined events from audio), but no benchmark evaluates joint audio-visual hallucination.
Approach
A benchmark with judgment tasks (audio-driven video hallucination, video-driven audio hallucination, audio-visual matching) and a description task (audio-visual captioning), built by disentangling in-view/out-of-view sound sources and sounds using GPT-4 with RAM++ tags and rule-based QnA generation, plus unimodal and text-only controls that diagnose the cause.
Key results
- Most audio-visual LLMs score near chance on AVHBench judgment tasks; e.g., Video-LLaMA is 50.1% accuracy on audio-driven video hallucination and PandaGPT reaches 58.5%.
- Unimodal inputs improve results: PLLaVA (video-only) reaches 81.5% accuracy versus near-chance for audio-visual models on the video hallucination task.
- Combining audio alignment with LoRA fine-tuning on Video-LLaMA raises audio-driven video accuracy to 83.9% and video-driven audio accuracy to 77.3%, from 50.1% and 50.2%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace simultaneous audio-visual inputs with unimodal inputs | Audio-visual LLMs · AVHBench Audio-driven Video and Video-driven Audio Hallucination | accuracy | Helped |
| Replace direct multimodal inputs with text captions of the audio and video | Audio-visual LLMs · AVHBench Audio-driven Video and Video-driven Audio Hallucination | accuracy | Helped |
| Add audio feature alignment (ablation) | Video-LLaMA · AVHBench Video-driven Audio Hallucination (multi.) | accuracy: 50.2 -> 58.1 | Helped |
| Add audio feature alignment (ablation) | Video-LLaMA · AVHBench Audio-driven Video Hallucination (multi.) | accuracy: 50.1 -> 52.8 | Helped |
| Add LoRA fine-tuning on the annotation-enriched dataset (ablation) | Video-LLaMA · AVHBench Audio-driven Video Hallucination (multi.) | accuracy: 50.1 -> 79.1 | Helped |
| Add LoRA fine-tuning on the annotation-enriched dataset (ablation) | Video-LLaMA · AVHBench Video-driven Audio Hallucination (multi.) | accuracy: 50.2 -> 76.6 | Helped |
| Add both audio feature alignment and LoRA fine-tuning (ablation) | Video-LLaMA · AVHBench Audio-driven Video Hallucination (multi.) | accuracy: 50.1 -> 83.9 | Helped |
| Add both audio feature alignment and LoRA fine-tuning (ablation) | Video-LLaMA · AVHBench Audio-visual Matching | accuracy: 50.0 -> 55.6 | Helped |
| Add audio feature alignment and LoRA fine-tuning (ablation) | Video-LLaMA · VAST and AVinstruct | CIDEr: 0.2 -> 47.6; 14.5 -> 102.0 | Mixed |
| Add audio feature alignment without LoRA fine-tuning (ablation) | Video-LLaMA · AVinstruct | METEOR: 45.9 -> 42.2 | Hurt |
Surprising findings
- Audio-visual LLMs perform better with unimodal or caption-based text-only inputs than with joint audio-visual input.
- Several audio-visual models answer almost always "Yes" or nearly always "No", giving high recall or low recall without real discrimination.
Limitations
- Focuses on environmental sounds rather than speech.
- Limited to 10-second clips and does not cover multi-object or temporal reasoning.
- Automatic annotation relies on GPT-4 and off-the-shelf tagging, with biases from AudioCaps and VALOR.
Experimental setup
Datasets: VALOR AudioCaps Clotho WavCaps VAST AVinstruct
Benchmarks: AVHBench VAST AVinstruct
Metrics: accuracy precision recall F1 yes ratio METEOR CIDEr GAVIE-A
Baselines: X-InstructBLIP ImageBind-LLM Video-LLaMA ChatBridge PandaGPT OneLLM Video-SALMONN Video-LLaMA2 Gemini-Flash PLLaVA
Builds on: POPE Video-LLaMA LoRA GAVIE RAM++
Closest papers by meaning
- SVHalluc: Benchmarking Speech-Vision Hallucination in Audio-Visual Large Language Models (CVPR 2026 · similarity 0.94)
- The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audio (NeurIPS 2025 · similarity 0.94)
- AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decoding (NeurIPS 2025 · similarity 0.94)
- MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimization (CVPR 2026 · similarity 0.93)
- Combating Multimodal LLM Hallucination via Bottom-Up Holistic Reasoning (AAAI 2025 · similarity 0.93)
- VidHalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understanding (CVPR 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.