Research map / Vision-language and multimodal / Multimodal LLMs

XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Models

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces XModBench, a 61,320-question tri-modal (audio, vision, text) multiple-choice benchmark that measures cross-modal consistency of omni-modal LLMs via six modality configurations, finding that even Gemini 2.5 Pro shows large audio disparities and directional imbalance.

Problem

Omni-modal LLMs are assumed to unify audio, vision, and text, but it is unclear whether they reason in a modality-invariant way or show modality-specific biases; existing benchmarks rarely test cross-modal consistency.

Approach

Builds each question as context plus candidates and permutes context and candidate modalities across text, vision, and audio to create six semantically identical configurations, then diagnoses task competence, modality disparity, and directional imbalance from accuracy differences.

Key results

• Gemini 2.5 Pro achieves the highest average accuracy of 70.6% across six modality configurations, but stays below 60% on spatial reasoning (50.1) and temporal reasoning (60.8) at the family level.
• Audio-involving configurations lag strongly: Gemini 2.5 Pro shows a text-vs-audio disparity of about -49 points, versus -15 for text-vs-vision.
• Directional imbalance is largest for vision-text pairs; Qwen2.5-Omni shows a 16.6-point gap between T->V and V->T.

Surprising findings

• Gemini 2.5 Pro reaches only about 50% on spatial reasoning, and EchoInk-R1 (fine-tuned from Qwen2.5-Omni) shows larger modality disparities than its base model.
• Adding audio and visual context together (A+V) gives only modest and inconsistent gains; Gemini 2.0 Flash drops by 5.36 points.

Limitations

• Closed-source Gemini models evaluated via API with unknown training data, limiting causal attribution of observed biases.
• Several benchmark components rely on synthetic audio (FireRedTTS) and GPT-5-refined question text, which may introduce artifacts.
• XModBench-Lite release is described as future work; main results use the full benchmark only.

Experimental setup

Datasets: VGG-Sound STARSS23 OmniInstruct Music AVQA

Benchmarks: XModBench XModBench-Lite

Metrics: accuracy standard deviation across modality configurations modality disparity directional imbalance

Baselines: Gemini 2.5 Pro Gemini 2.5 Flash Gemini 2.0 Flash Gemini 1.5 Pro Qwen2.5-Omni EchoInk-R1 Baichuan Omni 1.5 VideoLLaMA 2 VITA Unified-IO 2 PandaGPT Qwen2.5-VL

Builds on: MMBench MME OmniBench AV-Odyssey Bench

Closest papers by meaning

• OmnixR: Evaluating Omni-modality Language Models on Reasoning across Modalities (ICLR 2025 · similarity 0.93)
• OmniBench: Towards The Future of Universal Omni-Language Models (NeurIPS 2025 · similarity 0.92)
• Evaluating Cross-Modal Reasoning Ability and Problem Charactaristics with Multimodal Item Response Theory (ICLR 2026 · similarity 0.92)
• Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoning (ICLR 2026 · similarity 0.91)
• Measuring Cross-Modal Synergy: A Benchmark for VLM Explainability (NeurIPS 2026 · similarity 0.90)
• Diagnosing and Mitigating Modality Interference in Multimodal Large Language Models (NeurIPS 2026 · similarity 0.90)