Research map / Vision-language and multimodal / Multimodal LLMs

CrossCheck-Bench: Diagnosing Compositional Failures in Multimodal Conflict Resolution

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces CrossCheck-Bench, a hierarchical (L1-L3) benchmark of ~15k synthetic-contradiction e-commerce QA pairs with seven atomic capabilities, evaluating 13 VLMs and showing performance declines from perception to conflict reasoning, with a proposed MM-CoT prompting protocol giving modest gains.

Problem

Vision-language models are trained and evaluated mostly on aligned image-text pairs, leaving their ability to detect and resolve cross-modal contradictions in real-world content largely unexplored.

Approach

A diagnostic benchmark that decomposes cross-modal conflict resolution into seven atomic capabilities and three compositional levels (perception, integration, reasoning), built from e-commerce artifacts with injected contradictions, plus a two-stage interleaved prompting method (MM-CoT) that combines SoM grounding with reasoning feedback.

Key results

• Human upper bound averages 95.2% while the best proprietary model (GPT-4.1, 76.8%) trails by over 18 points, and open models peak at 71.5%.
• Accuracy declines from L1 to L3 across models; GPT-4.1 drops from 85.3% on L1 to 75.7% on L3, and InternVL3-78B from 71.5% to 64.0%.
• Adding capabilities to paired tasks lowers accuracy, e.g., BI* (A3,4,7) falls 30.6 points on average relative to the baseline, and MM-CoT yields the best prompting results (e.g., +4.4% for GPT-4o).

Surprising findings

• Chain-of-Thought and Set-of-Mark prompting yield only marginal gains and CoT can harm perception-centric tasks.
• Model scaling improves perception-level tasks but gives inconsistent or diminishing returns at the reasoning level.

Limitations

• Benchmark data is almost entirely e-commerce, limiting domain coverage.
• Labels and difficulty rely partly on GPT-4o and other model consensus, and GPT-4o is used as the open-ended judge, which may introduce bias.
• Reported numbers and internal details are inconsistent across text, tables and appendix (e.g., 15k vs 14.69k QA pairs, varying GPT-4.1 L1/L3 figures), reducing verifiability.

Experimental setup

Datasets: CrossCheck-Bench e-commerce listings (22.8k MCGs)

Benchmarks: CrossCheck-Bench

Metrics: accuracy exact match

Baselines: GPT-4.1 GPT-4.1-mini Gemini-2.5-pro ByteDance-seed-1.6 Qwen2.5-VL-72B InternVL3-78B Ovis2-34B MiMo-VL-7B Chain-of-Thought Set-of-Mark

Builds on: chain-of-thought set-of-mark prompting

Closest papers by meaning

• RPTS: Tree-Structured Reasoning Process Scoring for Faithful Multimodal Evaluation (AAAI 2026 · similarity 0.91)
• Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoning (ICLR 2026 · similarity 0.90)
• Evaluating Cross-Modal Reasoning Ability and Problem Charactaristics with Multimodal Item Response Theory (ICLR 2026 · similarity 0.90)
• PRISM-Bench: A Benchmark of Puzzle-Based Visual Tasks with CoT Error Detection (NeurIPS 2026 · similarity 0.90)
• XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Models (ICLR 2026 · similarity 0.90)
• LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.90)