Research map / Vision-language and multimodal / Multimodal LLMs

EduDiag: A Benchmark for Educational Diagnostic Reasoning with Error Tracing and Correction on Large Multimodal Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces EduDiag, a benchmark (8,345 erroneous reasoning chains with feedback across commonsense, science, math) for evaluating LMMs on error tracing and corrective feedback, and shows GRPO with an incorrect-answer reward improves open-source LMMs.

Problem

Existing multimodal reasoning benchmarks test forward chain-of-thought only and lack annotated erroneous reasoning chains and corrective feedback, so LMMs' ability to diagnose student errors (educational diagnostic reasoning) is under-explored.

Approach

Given question, image, rationale and an incorrect answer, the LMM must reconstruct the erroneous reasoning chain leading to that answer and produce corrective feedback; annotations are AI-assisted (GPT-4o) with human verification, and GRPO with an incorrect-answer-matching reward is used to improve error tracing.

Key results

• GPT-5 reaches only 52.22% overall Accp versus 62.67% for ground-truth feedback, and only 2.38 Se on mathematics.
• GRPO with the R2 incorrect-answer reward raises Qwen2.5-VL-7B overall Accp from 51.75% (SFT) to 53.46%, and Qwen3-VL-8B from 51.81% to 53.85%.
• Qwen2.5-VL-7B Se rises from 0.38 to 1.97 after SFT, but Accp gains are only about 1-2 points.

Surprising findings

• Proprietary GPT-5 and Gemini-2.5-Pro score lower on mathematics correction accuracy than some open-source models, with Gemini-2.5-Pro at 27.75% math Accp.
• SFT substantially improves erroneous-chain scores but yields only marginal correction accuracy gains.

Limitations

• Erroneous chains and feedback are generated by GPT-4o with human filtering, so the benchmark may inherit GPT-4o biases.
• Evaluation of Se and Sf relies on LLM judges, with possible judge bias despite using three judges.
• Source data limited to M3CoT commonsense/science and MATH-Vision math subsets (3,044 questions).

Experimental setup

Datasets: M3CoT MATH-Vision EduDiag

Benchmarks: EduDiag

Metrics: Erroneous Chains Score (Se) Feedback Score (Sf) Correction Accuracy (Accp) distractor accuracy

Baselines: GPT-4o GPT-5 Claude-3.5-Sonnet Gemini-2.5-Pro Qwen2.5-VL-7B InternVL3-8B Llama-3.2-90B GLM-4.1V-Thinking

Builds on: M3CoT MATH-Vision ROSCOE VISCO GRPO

Closest papers by meaning

• MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency (ICML 2025 · similarity 0.88)
• BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Dataset (NeurIPS 2025 · similarity 0.87)
• AStar: Boosting Multimodal Reasoning with Automated Structured Thinking (AAAI 2026 · similarity 0.86)
• MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models (AAAI 2026 · similarity 0.86)
• Conversational Learning Diagnosis via Reasoning Multi-Turn Interactive Learning (AAAI 2026 · similarity 0.86)
• Unveiling Fine-Grained Visual Traces: Evaluating MultiModal Interleaved Reasoning Chains in Multimodal STEM Tasks (NeurIPS 2026 · similarity 0.86)