Research map / Large language models / AI and society
OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AI
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OlympicArena, a bilingual, multimodal benchmark of 11,163 Olympic-level problems across seven disciplines, with answer-level and process-level evaluation of LLMs and LMMs, finding GPT-4o reaches only 39.97% overall accuracy.
Problem
Existing benchmarks rely on knowledge-intensive, mostly text-only exam problems that are now easy for LLMs, and they evaluate only final answers, missing cognitive reasoning quality and multimodal reasoning.
Approach
A large Olympic-competition benchmark spanning seven disciplines with text-only and interleaved text-image inputs, 13 answer types, fine-grained logical and visual reasoning labels, GPT-4V-annotated difficulty levels, and step-level process evaluation via GPT-4V scoring.
Key results
- GPT-4o achieves 39.97% overall accuracy on OlympicArena, while other open-source models stay below 20%.
- Most LMMs show little or negative gain from interleaved image inputs compared to their text-only counterparts; only GPT-4o and Qwen-VL-Chat show clear improvement.
- Process-level accuracy is often higher than answer-level accuracy, and model-based step scoring reached 83% inter-annotator agreement.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use interleaved image inputs instead of text-only inputs | GPT-4o · OlympicArena overall | accuracy: 38.17 -> 39.97 | Helped |
| Use image captions instead of text-only inputs | GPT-4o · OlympicArena overall | accuracy: 38.17 -> 38.50 | Helped |
| Use interleaved image inputs instead of text-only inputs | GPT-4 · OlympicArena overall | accuracy: 32.37 -> 32.76 | Helped |
| Use image captions instead of text-only inputs | GPT-4 · OlympicArena overall | accuracy: 32.37 -> 33.00 | Helped |
| Use interleaved image inputs instead of text-only inputs | Claude3 Sonnet · OlympicArena overall | accuracy: 23.02 -> 23.13 | No clear effect |
| Use image captions instead of text-only inputs | Claude3 Sonnet · OlympicArena overall | accuracy: 23.02 -> 23.10 | No clear effect |
| Use image captions instead of text-only inputs | Qwen-7B-Chat · OlympicArena overall | accuracy: 4.31 -> 4.89 | Helped |
| Use image captions instead of text-only inputs | Yi-34B-Chat · OlympicArena overall | accuracy: 17.31 -> 16.72 | Hurt |
| Use image captions instead of text-only inputs | InternLM2-20B-Chat · OlympicArena overall | accuracy: 16.62 -> 17.07 | Helped |
| Use image captions instead of text-only inputs | Qwen1.5-32B-Chat · OlympicArena overall | accuracy: 23.69 -> 23.43 | Hurt |
| Use image captions instead of text-only inputs | GPT-3.5 · OlympicArena overall | accuracy: 18.27 -> 18.83 | Helped |
| Use image captions instead of text-only inputs | GPT-4o · OlympicArena overall | Pass@1: 8.46 -> 9.57 | Helped |
Surprising findings
- Most LMMs do not benefit from image inputs and some perform worse than their text-only counterparts.
- Data leakage detection found very few leaked instances, and models rarely answer the leaked ones correctly.
Limitations
- Process-level evaluation uses only 96 sampled problems and GPT-4V as judge, with open-source model-based evaluation left for future work.
- Difficulty and reasoning-ability labels were produced by GPT-4V and only partly manually verified.
- Many models were evaluated only with zero-shot prompts and temperature 0 under a 2048-token output cap.
Experimental setup
Datasets: OlympicArena
Benchmarks: OlympicArena
Metrics: accuracy pass@1 process-level step correctness score
Baselines: GPT-4 Claude3 Sonnet Gemini Pro Vision Qwen-VL-Max LLaVA-NeXT-34B InternVL-Chat-V1.5 Yi-VL-34B Qwen-VL-Chat
Builds on: MMMU OlympiadBench MMCode
Closest papers by meaning
- Evaluating Large Vision-and-Language Models on Children's Mathematical Olympiads (NeurIPS 2024 · similarity 0.89)
- Creativity or Brute Force? Using Brainteasers as a Window into the Problem-Solving Abilities of Large Language Models (NeurIPS 2025 · similarity 0.88)
- Omni-MATH: A Universal Olympiad Level Mathematic Benchmark for Large Language Models (ICLR 2025 · similarity 0.88)
- Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzles (NeurIPS 2025 · similarity 0.87)
- PuzzleWorld: A Benchmark for Multimodal, Open-Ended Reasoning in Puzzlehunts (ICLR 2026 · similarity 0.87)
- Eliciting Reasoning in Language Models with Cognitive Tools (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.