Research map / Vision-language and multimodal / Multimodal LLMs

When Eyes and Ears Disagree: Can MLLMs Discern Audio-Visual Confusion?

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces AV-ConfuseBench to test whether MLLMs notice audio-absent but visually present objects, and RL-CoMM, a GRPO-based method on Qwen2.5-Omni-3B that uses an audio-only LALM reference reasoning plus answer-centered confidence optimization to reduce visual bias.

Problem

MLLMs rely on visual evidence and fail to recognize when an object visible in a video has no corresponding sound (audio-visual confusion), and their reasoning is biased toward visual content in audio-visual tasks.

Approach

Use an external large audio language model to generate audio-only reference reasoning, and reward the omni-model's audio reasoning for agreement with this reference (Step-wise Reasoning Reward) plus audio-visual coherence; then add an answer-centered NLL plus entropy-minimization loss to sharpen answer confidence.

Key results

• On AV-ConfuseBench audio-muted setting, RL-CoMM reaches 27.40% accuracy vs 8.22% for Qwen2.5-Omni-3B and 15.07% for GRPO.
• On Music-AVQA, RL-CoMM averages 79.46% vs 54.95% for Qwen2.5-Omni-3B and 70.05% for GRPO.
• On AVHBench audio-driven video hallucination, RL-CoMM reaches 78.96% accuracy vs 65.85% for Qwen2.5-Omni-3B.

Surprising findings

• Gemini 2.5 Pro with thinking mode still answers the muted-object question affirmatively in a large share of cases, and open-source models answer yes over 90% of the time.
• SFT on the warm-up data reduced accuracy on AV-ConfuseBench (8.22% to 5.48%).

Limitations

• Small benchmark: 39 videos and 73 audio-muted QA pairs, plus 20 videos and 100 audio-modified QA pairs.
• Reference reasoning depends on an external LALM whose identity is not specified in the text.
• Performance on audio-visual matching did not improve meaningfully and remains weak.

Experimental setup

Datasets: Music-AVQA AVQA AVHBench Music-AVQA-R

Benchmarks: AV-ConfuseBench Music-AVQA Music-AVQA-R AVQA AVHBench

Metrics: accuracy yes ratio precision F1 A-Acc. V-Acc.

Baselines: Qwen2.5-Omni-3B Qwen2.5-Omni-7B Gemini 2.5 Pro Gemini 2.5 Flash Video-LLaMA One-LLM PSTP-Net CAD AVHBench baselines (ChatBridge, PandaGPT, OneLLM)

Builds on: GRPO Qwen2.5-Omni Qwen3 Embedding

Closest papers by meaning

• HAVE-Bench: Hierarchical Audio-Visual Evaluation from Perception to Interaction (CVPR 2026 · similarity 0.92)
• OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.91)
• JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation (ICLR 2026 · similarity 0.90)
• MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix (NeurIPS 2025 · similarity 0.90)
• Mellow: a small audio language model for reasoning (NeurIPS 2025 · similarity 0.90)
• Hear, Localize, and Reason: Spatially Aware Scene Understanding for Audio-visual LLMs (NeurIPS 2026 · similarity 0.89)