Research map / Vision-language and multimodal / Cross-modal learning

The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audio

NeurIPS 2025 · Vision-language and multimodal › Cross-modal learning · approach: Audio-visual speech separation · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces CMM, a 1,200-sample yes/no probing benchmark for hallucinations in LMMs across visual, audio, and joint audio-visual-language inputs, attributing hallucinations to unimodal overreliance and spurious inter-modality correlations.

Problem

Hallucination in large multimodal models is well studied for image-text but largely unexplored when audio and video are added; existing benchmarks do not diagnose which cross-modal factors cause it.

Approach

Frames hallucination evaluation as binary yes/no existence probing at object and event granularity, organized into two causal factors (unimodal overreliance: visual, audio, language dominance; spurious inter-modality correlations: VL, AL, VAL), scored with perception accuracy and hallucination resistance.

Key results

• Gemini-1.5-pro reaches only 14.5 HR on Audio-Language correlations despite 94.0 PA.
• Gemini-1.5-flash scores 86.5 HR on Audio Dominance but only 36.5 HR on Visual Dominance.
• GroundingGPT answers yes indiscriminately, with 0.0 HR on Audio-Language correlations and 96.6 overall PA but 14.3 overall HR.

Measured effects reported in the paper

Surprising findings

• Larger LLM decoders improve language-dominance HR but barely change visual-language correlation HR, and GroundingGPT's very high PA paired with near-zero HR indicates indiscriminate yes answers.

Limitations

• Benchmark is small (1,200 samples, 200 per subcategory) and relies on manual curation.
• Evaluation uses only yes/no probing questions and a single decoding setting, which may not reflect open-ended hallucination.
• Ground truth for some subcategories depends on manual annotation and sources limited to AudioCaps, WebVid10M, and Auto-acd.

Experimental setup

Datasets: AudioCaps WebVid10M Auto-acd

Benchmarks: CMM

Metrics: perception accuracy (PA) hallucination resistance (HR)

Baselines: Gemini-1.5-pro Gemini-1.5-flash Reka-core FAVOR GroundingGPT VideoLLaMA2 GPT-4o VideoChat2 ShareGPT4Video PLLaVA CogVLM2-Video InternLM-XComposer2.5

Builds on: AudioCaps WebVid10M Auto-acd

Closest papers by meaning

• AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Models (ICLR 2025 · similarity 0.94)
• Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causality (ICLR 2025 · similarity 0.93)
• Robust Multimodal Large Language Models Against Modality Conflict (ICML 2025 · similarity 0.93)
• MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimization (CVPR 2026 · similarity 0.93)
• Combating Multimodal LLM Hallucination via Bottom-Up Holistic Reasoning (AAAI 2025 · similarity 0.92)
• Mitigating Multimodal Hallucinations via Gradient-based Self-Reflection (CVPR 2026 · similarity 0.91)