Research map / Vision-language and multimodal / Multimodal LLMs
RBench-V: A Primary Assessment for Visual Reasoning Models with Multimodal Outputs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces RBench-V, an 803-question benchmark that requires models to generate or manipulate images during visual reasoning (math, physics, counting, games), and evaluates MLLMs, omni-models, thinking models and human experts.
Problem
Existing multimodal benchmarks mostly test multimodal inputs with text-only outputs and text-only reasoning, leaving the ability to reason through generated or modified visual outputs (multimodal chain of thought) unmeasured.
Approach
A benchmark whose questions are designed so that solving them requires producing new visual content (drawing auxiliary lines, trajectories, connections, transformed figures) during reasoning, scored with an LLM-as-judge framework and compared against human experts.
Key results
- OpenAI o3 achieves the best overall accuracy of 25.8% on RBench-V, versus 82.3% for human experts.
- Open-source models score between about 7.7% and 10.6% overall; Qwen2.5VL-72B reaches 10.6%, with no clear gain from scaling Qwen2.5VL from 7B to 72B.
- Omni-models (Qwen2.5-Omni-7B 7.7%, MiniCPM-2.6-o 9.7%) and long text-only thinking models (Doubao-1.5-thinking-pro-m 17.1% vs. Doubao-1.5-vision-pro 15.6%) show only marginal gains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Scale Qwen2.5VL from 7B to 72B | Qwen2.5VL · RBench-V | accuracy (%): 8.3 -> 10.6 | No clear effect |
| Scale InternVL-3 from 8B to 38B | InternVL-3 · RBench-V | accuracy (%): 8.2 -> 10.0 | No clear effect |
| Scale LLaVA-OneVision from 7B to 72B | LLaVA-OneVision · RBench-V | accuracy (%): 8.5 -> 9.0 | No clear effect |
| Add omni-model image output decoding | Qwen2.5VL-7B · RBench-V | accuracy (%): 8.3 -> 7.7 | Hurt |
| Add omni-model image output decoding | MiniCPM-2.6-V · RBench-V | accuracy (%): 9.1 -> 9.7 | Helped |
| Use long text-only chain-of-thought reasoning instead of vision-pro | Doubao-1.5 · RBench-V | accuracy (%): 15.6 -> 17.1 | Helped |
| Use OpenAI o3 instead of the previous strongest model | RBench-V models · RBench-V | accuracy (%): +4.9% | Helped |
| Use OpenAI o3 instead of human experts | RBench-V evaluated systems · RBench-V overall | accuracy (%): 25.8% ... 82.3% | Hurt |
| Use open-source rather than closed-source models | RBench-V evaluated models · RBench-V overall | accuracy (%): 10.6% vs. 25.8% | Hurt |
Surprising findings
- Scaling model size from 7B to 72B (Qwen2.5VL) does not clearly improve performance on the benchmark.
- Models often convert geometry problems into algebraic coordinate-based solutions rather than the geometric drawing approach humans use, creating 'multimodal reasoning shortcuts'.
Limitations
- Relatively small benchmark (803 questions) across four domains.
- Relies on GPT-4o as LLM judge for both multiple-choice and open-ended answers.
- Only text and image modalities are covered; video and audio outputs are not included.
Experimental setup
Datasets: RBench-V
Benchmarks: RBench-V
Metrics: Top-1 accuracy
Baselines: o3 Gemini 2.5 pro GPT-4o Qwen2.5-VL Doubao-1.5-thinking-pro-m o4-mini o1 Claude-3.7-sonnet
Builds on: MMMU MMLU
Closest papers by meaning
- MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods (NeurIPS 2026 · similarity 0.92)
- Benchmarking Multimodal Mathematical Reasoning with Explicit Visual Dependency (NeurIPS 2026 · similarity 0.92)
- ROVER: Benchmarking Reciprocal Cross-Modal Reasoning for Omnimodal Generation (ICLR 2026 · similarity 0.91)
- CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Models (AAAI 2025 · similarity 0.91)
- RBench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation (ICML 2025 · similarity 0.91)
- LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.