Research map / Large language models / Agents, code and math
RBench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces RBench, a graduate-level, English-Chinese multi-disciplinary benchmark (1,094 text and 665 multimodal questions) for evaluating LLM and MLLM complex reasoning, and evaluates many proprietary and open models on it.
Problem
Existing reasoning benchmarks such as MMLU and MMMU are near saturation for frontier models and lack multimodal, multilingual, and multi-disciplinary difficulty at graduate level, while math olympiad benchmarks are narrow.
Approach
A graduate-level benchmark built from Tsinghua course materials, with expert screening, a filter keeping only questions where o1 uses at least 2,000 reasoning tokens, conversion to 6-option single-choice format, and parallel English-Chinese versions for both text-only and multimodal questions.
Key results
- o1-20241217 scores 69.0% on RBench-T and 53.2% on RBench-M, the best multimodal result reported.
- GPT-4o scores 53.6% on RBench-T but 33.4% on RBench-M, showing a large text-to-multimodal gap.
- In expert pairwise voting, RBench-T was judged to require more reasoning than MMLU in 85.94% of comparisons.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add CoT prompt (ablation) | GPT-4o-20241120 · RBench-T | accuracy (%): 51.5 -> 53.6 | Helped |
| add CoT prompt (ablation) | Llama-3.3-70B-Instruct · RBench-T | accuracy (%): 47.4 -> 49.5 | Helped |
| add CoT prompt (ablation) | Qwen2.5-32B-Instruct · RBench-T | accuracy (%): 44.6 -> 50.8 | Helped |
| add CoT prompt (ablation) | Qwen2.5-7B-Instruct · RBench-T | accuracy (%): 42.6 -> 43.6 | Helped |
| add CoT prompt (ablation) | o1-mini@20240912 · RBench-T | accuracy (%): 64.0 -> 64.0 | No clear effect |
| evaluate multimodal rather than text-only reasoning | o1-20241217 · RBench-T vs RBench-M | accuracy (%): 69.0 -> 53.2 | Hurt |
| evaluate multimodal rather than text-only reasoning | GPT-4o · RBench-T vs RBench-M | accuracy (%): 53.6% on text but only 33.7% in multimodal reasoning | Hurt |
| use RBench-T questions instead of MMLU questions | o1 · RBench-T vs MMLU | average thinking time: 13.5s -> 98.2s | Hurt |
| use RBench-M questions instead of MMMU questions | o1 · RBench-M vs MMMU | average thinking time: 20.3s -> 91.7s | Hurt |
Surprising findings
- Chain-of-thought prompting gives no gain for o1-mini (64.0% with and without CoT), suggesting reasoning models already perform internal CoT.
- Open-source models show a larger gap to closed models on multimodal reasoning than on text.
Limitations
- Proof-based questions were excluded because automated verification is not possible.
- Questions were filtered using o1 reasoning-token counts, which makes the benchmark dependent on one model's behavior.
- Subject coverage is drawn from a single university's curriculum (Tsinghua).
Experimental setup
Datasets: Tsinghua University course materials (exams, textbooks, quizzes, homework)
Benchmarks: RBench-T RBench-T(zh) RBench-M RBench-M(zh)
Metrics: top-1 accuracy consistency across languages thinking time (reasoning tokens)
Baselines: OpenAI o1 GPT-4o DeepSeek-R1 Gemini-2.0-flash-thinking Claude3.5-sonnet Qwen2.5-72B-Instruct Qwen2-VL-72B InternVL-2.5-8B MMLU MMMU
Builds on: MMLU MMMU chain-of-thought prompting
Closest papers by meaning
- MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models (AAAI 2026 · similarity 0.92)
- RBench-V: A Primary Assessment for Visual Reasoning Models with Multimodal Outputs (NeurIPS 2025 · similarity 0.91)
- BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Dataset (NeurIPS 2025 · similarity 0.91)
- MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency (ICML 2025 · similarity 0.90)
- LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.90)
- RPTS: Tree-Structured Reasoning Process Scoring for Faithful Multimodal Evaluation (AAAI 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.