Research map / Vision-language and multimodal / Multimodal LLMs

HAVE-Bench: Hierarchical Audio-Visual Evaluation from Perception to Interaction

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces HAVE-Bench, a 2,451-sample hierarchical audio-visual benchmark (Perception, Reasoning, Interaction) that evaluates omni MLLMs and finds large open-closed gaps, speech-image underperformance versus text-image, and poor multi-turn interaction.

Problem

Existing audio-visual benchmarks focus on passive perception and fragmented dimensions, lacking a unified hierarchical evaluation that covers audio-guided reasoning and memory-dependent multi-turn interaction.

Approach

A three-level cognitive hierarchy (perception, reasoning, interaction) crossed with two audio roles (audio-as-instruction and audio-as-context), including multi-turn task-graph interaction scenarios (navigation, puzzle solving, music reproduction) judged by an LLM against per-edge success criteria.

Key results

• Gemini2.5-Flash is the strongest model with L1 68.9, L2 60.8, L3 30.4 average; no model exceeds 40% on interaction tasks.
• Open-closed gap widens from ~0.5 on Perception to ~8.2 on Reasoning and ~12.4 on Interaction versus Qwen2.5-Omni.
• All tested models score higher in text-image than speech-image mode on audio-as-instruction tasks, e.g., Gemini2.5-Flash 73.2 TI vs 71.3 SI.

Surprising findings

• Qwen2.5-Omni surpasses Gemini2.5-Flash on some perception subtasks such as Instructed Recognition and Cross-modal Matching.
• ASR-transcribed speech fed to text-image pipelines matches TI performance, implying the gap stems from speech-image reasoning transfer rather than content loss.

Limitations

• Interaction tasks are small (263 samples) and rely on LLM judging of multi-turn transitions.
• Navigation street-view images are licensed, so only panorama IDs and coordinates are released.
• Ola could not be run on multi-turn interaction tasks, leaving its interaction scores missing.

Experimental setup

Datasets: TextVQA ChartQA AudioSet FAVDBench ObjectFolder Music-AVQA PDMX M3AV Epic-Kitchen VGGSound

Benchmarks: HAVE-Bench

Metrics: accuracy success rate

Baselines: Gemini2.5-Flash Qwen2.5-Omni Ming-Lite-Omni MiniCPM-o Ola VITA-1.5 Megrez-3B-Omni GPT-4.1

Builds on: MMMU AudioSet Music-AVQA AVQA

Closest papers by meaning

• JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation (ICLR 2026 · similarity 0.93)
• AVI-Bench: Toward Human-like Audio-Visual Intelligence of Omni-MLLMs (ICML 2026 · similarity 0.92)
• When Eyes and Ears Disagree: Can MLLMs Discern Audio-Visual Confusion? (AAAI 2026 · similarity 0.92)
• OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.91)
• MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix (NeurIPS 2025 · similarity 0.90)
• LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.90)