Research map / Vision-language and multimodal / Multimodal LLMs

Unveiling Fine-Grained Visual Traces: Evaluating MultiModal Interleaved Reasoning Chains in Multimodal STEM Tasks

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces STEPSTEM, a 283-problem graduate-level multimodal STEM benchmark with complementary text-vision inputs and a step-level, multi-reference evaluation protocol for interleaved text-image reasoning, showing top MLLMs reach only 38.29% accuracy.

Problem

Existing multimodal STEM benchmarks permit unimodal shortcuts due to modality redundancy and score only final answers, so they cannot assess whether MLLMs perform genuine cross-modal reasoning in intermediate steps.

Approach

A benchmark whose problems enforce strict text-vision complementarity, annotated with multiple expert reference trajectories of interleaved text and image steps (with bounding boxes), scored by a fused local step-alignment (attention-rollout text matching, patch-based visual matching with diversity penalty, dynamic-programming alignment) and LLM-judge global coverage metric, taking the best over r

Key results

• Gemini 3.1 Pro and Claude Opus 4.6 reach only 38.29% text-answer accuracy on STEPSTEM, the best among evaluated models.
• Gemini 2.5 Flash Image achieves 50.80% image-answer accuracy on the 14 drawing tasks, the highest image score.
• GPT-5.4 + GPT-Image obtains the best fused process score of 71.52, versus 37.87-38.13 for the best understanding-only models.

Surprising findings

• Understanding-only models score zero on all image-based metrics, yet top text models still achieve the highest text-answer accuracy on the Mathematics subset.
• Interleaved generation models like Bagel and OmniGen2 get non-trivial image process scores but 0% image-answer accuracy.

Limitations

• Small benchmark of 283 problems, with only 14 image-answer drawing tasks and 30 mathematics problems.
• Evaluation relies on a Qwen3.5-9B judge and ResNet-50 visual features, whose reliability for STEM content is not fully validated.
• Human evaluation covers only 20 examples across four models, and the authors note that compressed or semantically equivalent reasoning paths are scored imperfectly.

Experimental setup

Datasets: STEPSTEM (283 expert-curated problems from public competitions, graduate textbooks, open educational resources)

Benchmarks: STEPSTEM

Metrics: accuracy fused process score local step alignment score global logic coverage score human step-prefix score

Baselines: Gemini 2.5 Flash Image GPT-5.4 + GPT-Image Bagel Janus-Pro OmniGen2 Showo2 Gemini 3.1 Pro Claude Opus 4.6 Qwen3.5-122B Qwen3.5-Plus Qwen3-VL-235B

Builds on: chain-of-thought PRM800K Zebra-CoT MMMU

Closest papers by meaning

• ROVER: Benchmarking Reciprocal Cross-Modal Reasoning for Omnimodal Generation (ICLR 2026 · similarity 0.91)
• RPTS: Tree-Structured Reasoning Process Scoring for Faithful Multimodal Evaluation (AAAI 2026 · similarity 0.91)
• MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods (NeurIPS 2026 · similarity 0.90)
• AStar: Boosting Multimodal Reasoning with Automated Structured Thinking (AAAI 2026 · similarity 0.90)
• TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding (ICML 2026 · similarity 0.90)
• When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for Visual Chain-of-Thought (CVPR 2026 · similarity 0.90)