Research map / Vision-language and multimodal / Multimodal LLMs
OmnixR: Evaluating Omni-modality Language Models on Reasoning across Modalities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OmnixR, an evaluation suite with a synthetic set (Omnify!, text rendered to image/audio/video from MMLU-Pro) and a real-world YouTube set, to measure reasoning consistency of omni-modality language models across modalities and mixtures.
Problem
Existing benchmarks cover at most two modalities and do not measure whether omni-modality language models reason consistently when the same question is presented in text, image, audio, video, or mixed-modality form.
Approach
Omnify! converts text questions into other modalities programmatically (rendered images, TTS audio, word-per-frame videos) while keeping information identical, enabling controlled cross-modal comparisons; it also introduces Extract-Then-Answer prompting that first transcribes the input and then reasons over the text.
Key results
- On OmnixR-Synth, Gemini-1.5-Pro drops from 77.5 (text) to 57.3 (image) and 36.3 (video) with CoT; ETA prompting raises Pro image accuracy to 73.5.
- Claude-3-Opus image accuracy rises from 26.9 (CoT) to 62.6 with ETA prompting.
- Mixed Video+Audio input reduces Gemini-1.5-Pro accuracy to 40.1 and Flash to 25.9 versus 77.5 and 69.9 for text.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Extract-Then-Answer prompting instead of direct CoT on synthetic images | OmnixR synthetic evaluation across models · OmnixR synthetic (MMLU-Pro), image | accuracy: 57.3 -> 73.5; 36.3 -> 68.1; 26.9 -> 62.6; 18.8 -> 48.1; 9.9 -> 43.2; 60.1 -> 66.7; 48.5 -> 58.4 | Helped |
| Use Extract-Then-Answer prompting instead of direct CoT on synthetic video | Gemini-1.5-Pro, Gemini-1.5-Flash, GPT-4o-mini · OmnixR synthetic (MMLU-Pro), video | accuracy: 36.3 -> 48.6; 15.1 -> 42.8; 18.6 -> 59.3 | Mixed |
| Use Extract-Then-Answer prompting instead of direct CoT on synthetic audio | Gemini-1.5-Pro and Gemini-1.5-Flash · OmnixR synthetic (MMLU-Pro), audio | accuracy: 56.6 -> 69.9; 53.9 -> 63.6 | Helped |
| Use Extract-Then-Answer prompting instead of direct CoT on real-world audio | Gemini-1.5-Pro and Gemini-1.5-Flash · OmnixR real-world, audio | accuracy: 71 -> 55; 64 -> 51 | Hurt |
| Use Extract-Then-Answer prompting instead of direct CoT on real-world video | Gemini-1.5-Pro, Gemini-1.5-Flash, GPT-4o, GPT-4o-mini · OmnixR real-world, video | accuracy: 64 -> 71; 53 -> 73; 73 -> 66; 66 -> 63 | Mixed |
| Use a better-format image instead of the original image format (ablation) | All tested models with CoT · OmnixR synthetic (MMLU-Pro), image | accuracy: 57.3 -> 64.6; 36.3 -> 43.6; 26.9 -> 33.5; 18.8 -> 28.9; 9.9 -> 19.1; 60.1 -> 65.5; 48.5 -> 52.1 | Helped |
| Use a better-format image instead of the original image format (ablation) | All tested models with ETA · OmnixR synthetic (MMLU-Pro), image | accuracy: 68.7 -> 73.5; 61.3 -> 68.1; 36.4 -> 62.6; 26.6 -> 48.1; 24.9 -> 43.2; 66.7 -> 66.9; 58.4 -> 61.7 | Helped |
| Increase the number of words shown per video frame (ablation) | Gemini-1.5-Pro and Gemini-1.5-Flash · OmnixR synthetic (MMLU-Pro), video | accuracy | Helped |
Surprising findings
- ETA prompting helps less on the real-world set, with audio performance dropping relative to CoT for both Gemini models.
- GPT-4o could not extract text from video in over 46.8% of samples, so its ETA video scores collapsed.
Limitations
- Synthetic conversion is simple (rendered text, TTS), so it may not reflect natural image/audio/video distributions.
- Real set is small (100 videos) and annotated by a few humans; the paper also notes MMLU-Pro and API version dependence.
- Open-source omni models were not evaluated because few support all modalities.
Experimental setup
Datasets: MMLU-Pro ARC-Challenge GSM8K
Benchmarks: OmnixR-Synth (MMLU-Pro derived) OmnixR-Real ARC-Challenge GSM8K
Metrics: accuracy reasoning path rate character error rate
Baselines: Gemini-1.5-Pro Gemini-1.5-Flash GPT-4o GPT-4o-mini Claude-3-Opus Claude-3-Sonnet Claude-3-Haiku
Builds on: MMLU-Pro chain-of-thought prompting
Closest papers by meaning
- XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Models (ICLR 2026 · similarity 0.93)
- OmniBench: Towards The Future of Universal Omni-Language Models (NeurIPS 2025 · similarity 0.93)
- ThinkOmni: Lifting Textual Reasoning to Omni-modal Scenarios via Guidance Decoding (ICLR 2026 · similarity 0.91)
- Evaluating Cross-Modal Reasoning Ability and Problem Charactaristics with Multimodal Item Response Theory (ICLR 2026 · similarity 0.91)
- OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.90)
- A Benchmark for Omni-Modal Reasoning in Long Videos (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.