Research map / Vision-language and multimodal / Multimodal LLMs

RPTS: Tree-Structured Reasoning Process Scoring for Faithful Multimodal Evaluation

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RPTS, a tree-structured, height-weighted faithfulness metric for multimodal reasoning chains, and RPTS-Eval, a 390-instance benchmark with 374 images, to catch right answers reached via flawed reasoning.

Problem

Existing multimodal benchmarks grade only final answers (or only check reasoning when answers are wrong), so correct answers reached through flawed reasoning go undetected, and cross-modal reasoning structure is not modeled.

Approach

Organize reasoning steps into a tree whose leaves are visual/textual clues, score each inference step with an LLM scorer, and aggregate with height-based exponential decay weights (controlled by lambda and h_f) into a single RPTS score that also localizes errors.

Key results

• GPT-4o achieves the highest English RPTS (0.84) and accuracy (0.86) on RPTS-Eval, with filtered accuracy 0.84 (only -0.02 drop).
• Open-source models show larger accuracy drops after RPTS filtering, e.g. Llava-Next-13B English accuracy falls from 0.62 to 0.34 under filtering.
• GPT-4 as scorer achieves the lowest mean absolute error (0.095) against human scores among the tested LLM scorers.

Surprising findings

• Open-source LVLMs keep relatively high RPTS scores despite low accuracy, which the authors attribute to logically consistent but off-target or repetitive reasoning.
• Chinese performance of several open-source LVLMs is markedly lower than English (e.g. Llava-Next-7B accuracy 0.13 vs 0.62).

Limitations

• Relies on GPT-4 both for reformatting reasoning and as the step scorer, so scores depend on a closed judge.
• Benchmark is small (390 instances) and mostly guided relationships (84.62%), limiting coverage of adversarial and independent cases.
• Hyperparameters lambda=0.9 and h_f=1 are chosen by design intuition and the sensitivity analysis is limited to one model.

Experimental setup

Datasets: RPTS-Eval

Benchmarks: RPTS-Eval

Metrics: accuracy RPTS filtered accuracy mean absolute error vs human scores

Baselines: InfiMM-Eval MMMU MM-Vet ROSCOE ReCEval

Builds on: chain-of-thought MM-Vet InfiMM-Eval

Closest papers by meaning

• LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.93)
• ROVER: Benchmarking Reciprocal Cross-Modal Reasoning for Omnimodal Generation (ICLR 2026 · similarity 0.92)
• MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods (NeurIPS 2026 · similarity 0.92)
• ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoning (ICLR 2026 · similarity 0.91)
• AStar: Boosting Multimodal Reasoning with Automated Structured Thinking (AAAI 2026 · similarity 0.91)
• Unveiling Fine-Grained Visual Traces: Evaluating MultiModal Interleaved Reasoning Chains in Multimodal STEM Tasks (NeurIPS 2026 · similarity 0.91)