Research map / Vision-language and multimodal / Multimodal LLMs

Omni-Captioner: Data Pipeline, Models, and Benchmark for Omni Detailed Perception

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Interleaved image-text generation with MLLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Omni-Detective, an agentic tool-calling data pipeline for detailed audio/audio-visual captions, trains Audio-Captioner and Omni-Captioner (Qwen2.5-Omni-7B based), and proposes Omni-Cloze, a cloze-style benchmark for omni detailed captioning.

Problem

Detailed captioning by omni language models is limited: longer captions gain detail but also hallucinate more, and existing evaluations (e.g., VDC's multi-QA) are costly and unstable, with no dedicated audio-visual detailed captioning benchmark.

Approach

An agentic Detective Agent iteratively queries tool-equipped observers (MLLMs, OCR, ASR) over multiple rounds to gather and cross-check grounded details, producing detailed captions with less hallucination; captioners are trained on this data in a two-stage curriculum, and a single-call cloze-with-"Not Given" benchmark separates omission from hallucination.

Key results

• Audio-Captioner reaches 70.0 on MMAU via caption-to-QA cascade, matching Gemini 2.5 Pro and best among open-source models.
• Omni-Captioner-7B reaches 55.0% accuracy and score 2.7 on VDC, a new state of the art, and 56.4% total on Omni-Cloze versus 43.6% for Gemini 2.5 Pro.
• Omni-Cloze correlates with human Elo ratings at Pearson r=0.91, higher than VDC and video-SALMONN 2 test metrics.

Surprising findings

• Detail and hallucination rise together as captions grow longer ("co-growth") in unconstrained detailed captioning by Gemini 2.5 Pro.
• Removing textual prompts from the input improved the model's descriptive performance.

Limitations

• Hallucination rate converges around step 5-6 of Omni-Detective, indicating a ceiling on tool-based error correction.
• Omni-Captioner lags on long videos in Video-MME (58.0%) and is below proprietary models in absolute terms.
• Cascade evaluation depends on GPT-4o as QA backbone and captions only proxy completeness.

Experimental setup

Datasets: VGGSound FineVideo MMAU MMAR VDC video-SALMONN 2 testset Video-MME Video-Holmes WorldSense Daily-Omni

Benchmarks: VDC video-SALMONN 2 testset MMAU MMAR Video-MME Video-Holmes WorldSense Daily-Omni Omni-Cloze

Metrics: accuracy score miss rate hallucination rate Elo score Pearson correlation

Baselines: Gemini 2.5 Pro Gemini 2.5 Flash GPT-4o video-SALMONN 2 Qwen2.5-Omni VideoLLaMA2 VideoLLaMA3 Qwen2-Audio MiDashengLM SALMONN

Builds on: Qwen2.5-Omni VDC video-SALMONN 2

Closest papers by meaning

• OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioning (NeurIPS 2026 · similarity 0.91)
• AVI-Bench: Toward Human-like Audio-Visual Intelligence of Omni-MLLMs (ICML 2026 · similarity 0.91)
• Omni-Interactive Universal Embedder (NeurIPS 2026 · similarity 0.90)
• OmniBench: Towards The Future of Universal Omni-Language Models (NeurIPS 2025 · similarity 0.89)
• OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.89)
• JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation (ICLR 2026 · similarity 0.88)