Research map / Vision-language and multimodal / Multimodal LLMs

FAVE: A Structured Benchmark for Fine-Grained Audio-Visual Temporal Evaluation in Multimodal LLMs

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces FAVE, a three-level benchmark (alignment, event-level relations, moment captioning) built on QVHighlights to evaluate how well audio-visual LLMs perform fine-grained temporal reasoning, finding that current AVLLMs struggle with joint audio-visual temporal understanding.

Problem

Audio-visual LLMs are not well evaluated on fine-grained temporal relationships between audio and visual streams, including cross-modal alignment, event ordering/proximity/position, and moment-level description.

Approach

A structured, multi-level benchmark (FAVE-align, FAVE-low, FAVE-high) built with a shot-segmentation plus captioning plus GPT-assisted annotation pipeline and manual verification, with speech as the primary audio modality and about 20% manually annotated environmental sounds, to test joint audio-visual temporal reasoning.

Key results

• Gemini 1.5 Flash achieves the top FAVE-align average of 4.04 out of 5 versus 4.63 for humans, and 80% order accuracy on FAVE-low.
• Gemini 1.5 Pro reaches 86.80% accuracy on event position, while the best FAVE-low average is 75.34% (Gemini 1.5 Flash) versus 88.98% for humans.
• On FAVE-high, all models score below 3 out of 5 and no open-source model achieves effective audio-visual integration; increasing sampled frames does not improve performance.

Surprising findings

• Increasing the number of sampled frames does not improve temporal awareness, suggesting information loss occurs on the model side rather than the input side.
• Adding audio to vision-language models often harms moment retrieval performance.

Limitations

• Speech is the primary audio modality, with environmental sounds only about 20% of the data.
• Human baseline is limited by caption ambiguity and annotator inconsistencies.
• Evaluation relies on GPT-based scoring for open-ended tasks, with a reported 5-point scale and GPT-human agreement above 95% on only 50 samples.

Experimental setup

Datasets: QVHighlights

Benchmarks: FAVE

Metrics: 5-point GPT similarity score accuracy 5-point relevance (visual/audio/overall)

Baselines: Gemini 1.5 Pro Gemini 1.5 Flash NextGPT Video-LLaMA PandaGPT VideoLLaMA2 ChatBridge OneLLM video-SALMONN TimeChat VTimeLLM LongVALE

Builds on: QVHighlights TransNetV2 Whisper 3D-Speaker

Closest papers by meaning

• Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding (AAAI 2025 · similarity 0.90)
• Aligned Better, Listen Better for Audio-Visual Large Language Models (ICLR 2025 · similarity 0.89)
• JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation (ICLR 2026 · similarity 0.89)
• OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioning (NeurIPS 2026 · similarity 0.89)
• Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · similarity 0.89)
• PreFM: Online Audio-Visual Event Parsing via Predictive Future Modeling (NeurIPS 2025 · similarity 0.88)