Research map / Vision-language and multimodal / Multimodal LLMs

OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioning

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces OmniCapBench, a test-only benchmark of 786 densely annotated videos that evaluates audio-visual captions via atomic, verifiable reference/shot/event units scored with deterministic rules plus bounded local LLM semantic checks.

Problem

Existing audio-visual caption evaluations either give whole-caption scores without localization, or local QA probes without coverage, and unconstrained LLM judges are unstable. Fine-grained diagnosis of identity tracking, temporal grounding and audio-visual association is therefore unreliable.

Approach

Models output native structured JSON (references, shots, events with time ranges and cross-links) instead of free-form captions; deterministic rules and IoU/WER matching verify structure, and LLMs only judge semantic equivalence on aligned unit pairs, with bidirectional precision/recall scoring.

Key results

• Gemini 2.5-Pro achieves 37.81% Cross-Shot Coreference Consistency, and open-source models fall below 12% CCC.
• Gemini 3.1-Pro drops from 63.32% Event F1 to 51.46% EVSA F1, with open-source models below 20% EVSA F1.
• Holistic judge scores for identical outputs range from 71.24 (weak judge) to 93.92 (strong judge), while OmniCapBench varies only from 56.82 to 58.24 across judges.

Surprising findings

• Seed2.0 emits almost no audio-event units (4.54% Event F1, 3.52% EVSA F1) despite strong speaker attribution (91.80%).
• MiniCPM-o-2.6 tops Scene precision (76.32%) while recalling only 20.39% of scene content.

Limitations

• Specialized video-captioning models were excluded from main evaluation due to poor instruction following on structured output.
• Reliance on frontier models in the annotation pipeline for generation and evaluation of the reference data.

Experimental setup

Datasets: UGC-VideoCap video-SALMONN 2

Benchmarks: OmniCapBench

Metrics: Ref Subject F1 Ref Scene F1 Shot F1 Subshot F1 tIoU RefUse CCC Event F1 Speaker F1 EVSA F1

Baselines: AuroraCap VCapsBench UGC-VideoCap video-SALMONN 2 Omni-Cloze LongVALE TimeChat OmniScript ASID-Caption AVoCaDO

Builds on: MTSS (Multi-Stream Scene Script) UGC-VideoCap Omni-Cloze

Closest papers by meaning

• JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation (ICLR 2026 · similarity 0.92)
• OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.92)
• Omni-Captioner: Data Pipeline, Models, and Benchmark for Omni Detailed Perception (ICLR 2026 · similarity 0.91)
• CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness (NeurIPS 2025 · similarity 0.90)
• MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos (NeurIPS 2026 · similarity 0.89)
• HAVE-Bench: Hierarchical Audio-Visual Evaluation from Perception to Interaction (CVPR 2026 · similarity 0.89)