Research map / Vision-language and multimodal / Multimodal LLMs

Evaluating Cross-Modal Reasoning Ability and Problem Charactaristics with Multimodal Item Response Theory

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes M3IRT, a multimodal multidimensional IRT that decomposes model ability and item difficulty into image-only, text-only and cross-modal components to select compact, cross-modal-demanding benchmark subsets.

Problem

Multimodal benchmarks contain shortcut questions solvable from a single modality, which inflate size and cost and yield unreliable model rankings; classical IRT cannot tell whether success reflects cross-modal reasoning.

Approach

Extends IRT/MIRT by decomposing ability and difficulty/discrimination into base, image, text and cross-modal terms, with the response under each input format (image/text presence) modeled by a 2PL-style sigmoid; fitted by SGD on partially observed responses and used with D-optimal CAT item selection.

Key results

• On MMMU, M2IRT reaches Spearman rank correlation 0.96 with the original ranking using only 10% of items, versus 0.82 for Random at 30%.
• On MathVista, M3IRT achieves 0.92 rank correlation at 10% of items, while FlashEval reaches 0.89 and Random 0.89.
• Low-quality question share in the selected subset for MMMU at 10% is 0.038 (M2IRT) and 0.091 (M3IRT), versus 0.31 for Random and 0.32 for IRT.

Surprising findings

• On VQA-AnswerTherapy, M2IRT and M3IRT produce lower rank correlation than baselines despite selecting fewer low-quality questions, which the authors attribute to the benchmark itself containing many low-quality items.
• Crowdsourced human answer patterns correlate weakly or negatively with estimated cross-modal difficulty on MMMU and MathVista, suggesting VLMs and humans exhibit different cross-modal behavior.

Limitations

• Restricted to multiple-choice questions; open-ended settings not studied.
• Low-quality questions are synthetically generated by swapping image or text, not realistic contamination.
• On VQA-AnswerTherapy the method underperforms baselines in rank correlation.

Experimental setup

Datasets: MMMU MathVista SEED-Bench VQA-AnswerTherapy

Benchmarks: MMMU MathVista SEED-Bench VQA-AnswerTherapy

Metrics: Spearman rank correlation proportion of low-quality questions in subset ROC-AUC

Baselines: Random IRT MIRT TinyBenchmarks FlashEval

Builds on: item response theory multidimensional IRT computerized adaptive testing TinyBenchmarks FlashEval

Closest papers by meaning

• XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Models (ICLR 2026 · similarity 0.92)
• LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.91)
• OmnixR: Evaluating Omni-modality Language Models on Reasoning across Modalities (ICLR 2025 · similarity 0.91)
• Measuring Cross-Modal Synergy: A Benchmark for VLM Explainability (NeurIPS 2026 · similarity 0.91)
• Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoning (ICLR 2026 · similarity 0.91)
• RPTS: Tree-Structured Reasoning Process Scoring for Faithful Multimodal Evaluation (AAAI 2026 · similarity 0.90)