Research map / Vision-language and multimodal / Multimodal LLMs
OmniBench: Towards The Future of Universal Omni-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OmniBench, a 1,142-question human-annotated benchmark requiring joint image, audio, and text reasoning, plus OmniInstruct (84.5K training samples) for tri-modal instruction tuning of omni-language models.
Problem
Existing multimodal benchmarks evaluate only one or two modalities, so there is no way to test whether models can jointly recognize, interpret, and reason over image, audio, and text where all three are needed for the correct answer.
Approach
A benchmark whose annotation pipeline enforces that each question needs both image and audio (verified by ablations with text-only, image-removed, and audio-removed inputs), with human rationales, plus a tri-modal instruction-tuning set derived from MSRVTT-QA, AVQA, and Music-AVQA2.0 and filtered by a VLM.
Key results
- Qwen-2.5-Omni achieves the top open-source overall accuracy of 56.13% on OmniBench, versus 42.91% for Gemini-1.5-Pro and 63.19% for human evaluators.
- Most open-source omni-language baselines score between roughly 18% and 38%, with several near random chance (25%).
- LoRA fine-tuning MiniCPM-o-2.6 on OmniInstruct raises OmniBench accuracy from 40.5% to 45.9%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune MIO-Instruct on 6.4K OmniInstruct samples with vanilla multimodal tokenizers | MIO-Instruct (7B) · OmniBench | accuracy: 24.8% -> 25.7% | Helped |
| filter non-speech tokens from the audio RVQ tokenizer during fine-tuning (ablation) | MIO-Instruct-Omni-V1 · OmniBench | accuracy: 25.7% -> 29.2% | Helped |
| LoRA fine-tune on full OmniInstruct | MiniCPM-o 2.6 · OmniBench | accuracy: 40.5% -> 45.9% | Helped |
| provide both image caption and audio transcript instead of either textual representation alone | Qwen2-Audio-7B-Instruct · OmniBench | accuracy: 39.05% and 39.67% -> 47.02% | Helped |
| replace raw audio with its transcript while retaining the image caption | Qwen2-Audio-7B-Instruct · OmniBench | accuracy: 40.72% -> 47.02% | Helped |
| replace image and audio with image caption and audio transcript | Claude-3.5-Sonnet · OmniBench | accuracy: 59.37% -> 56.83% | No clear effect |
| replace image and audio with image caption and audio transcript | GPT-4o (0513) · OmniBench | accuracy: 57.62% -> 60.51% | No clear effect |
| replace raw audio with its transcript | UnifiedIO2 models · OmniBench | accuracy: average gain 6.42% | Helped |
| replace the image with a generated caption | UnifiedIO2 models · OmniBench | accuracy: average performance drop 1.87% | Hurt |
| replace both image and audio with textual alternatives | UnifiedIO2 models · OmniBench | accuracy: average performance drops 0.12% | No clear effect |
| increase UnifiedIO2 model size from xlarge to xxlarge | UnifiedIO2 · OmniBench | accuracy: 38.00% -> 33.98% | Hurt |
| remove the image input (ablation) | Gemini-1.5-Pro · OmniBench | accuracy: 42.91% -> 27.93% | Hurt |
Surprising findings
- Text-only approximations (image caption plus audio transcript) improved accuracy for many models over their native image+audio inputs, e.g., Qwen2-Audio-7B-Instruct rose from 40.72% to 47.02%.
- Several open-source OLMs perform near or below random guessing (25%), e.g., Mu-LLaMA scores 1.58% and AnyGPT 18.04% in the text-approximation setting.
Limitations
- Only 1,142 questions (the appendix mentions 1,000 curated examples), limiting statistical power and coverage.
- Text-alternative image captions are LLM-generated and weak on OCR-style questions.
- Covers only static images, not video with audio.
Experimental setup
Datasets: MSRVTT-QA AVQA Music-AVQA2.0 OmniBench
Benchmarks: OmniBench
Metrics: accuracy
Baselines: Qwen-2.5-Omni MIO-Instruct AnyGPT Video-SALMONN VITA UnifiedIO2 Gemini-1.5-Pro Reka-core MiniCPM-o 2.6 Baichuan-Omni-1.5
Builds on: MSRVTT-QA AVQA Music-AVQA2.0 MIO-Instruct
Closest papers by meaning
- OmnixR: Evaluating Omni-modality Language Models on Reasoning across Modalities (ICLR 2025 · similarity 0.93)
- XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Models (ICLR 2026 · similarity 0.92)
- OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.92)
- MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos (NeurIPS 2026 · similarity 0.90)
- A Benchmark for Omni-Modal Reasoning in Long Videos (NeurIPS 2026 · similarity 0.90)
- OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-time Emotional Speech Synthesis (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.