Research map / Large language models / AI and society
SciArena: An Open Evaluation Platform for Non-Verifiable Scientific Literature-Grounded Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SciArena, a community-voting arena for literature-grounded scientific tasks with a Bradley-Terry leaderboard over 47 models, plus SciArena-Eval, a meta-evaluation benchmark for model-based judges.
Problem
Evaluating foundation models on open-ended, non-verifiable scientific literature-grounded tasks is hard: static expert benchmarks are small and quickly outdated, and LLM-based judges often disagree with domain experts.
Approach
Adapts the Chatbot Arena community-voting approach to scientific literature QA with domain-expert voters, a ScholarQA-based retrieval pipeline giving both models the same 30 paper contexts, and a meta-benchmark built from votes to test automated judges.
Key results
- o3 is top-ranked overall (Elo 1151.4), followed by Claude-4.1-Opus (1126.4) and GPT-5 (1122.5); open-source GPT-OSS-120B and DeepSeek-R1-0528 rank 8th and 9th.
- The best automated judge on SciArena-Eval, o3 with pairwise comparison, reaches 65.1% accuracy against human votes; several open models perform near random (50%).
- Response length coefficient of 0.141 is lower than Chatbot Arena (0.25), Vision Arena (0.27) and Search Arena (0.33), and citation-count coefficient is 0.039 versus 0.209 in Search Arena.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use o4-mini rather than GPT-4.1 as the pairwise evaluator | SciArena-Eval pairwise evaluation · SciArena-Eval | accuracy: 61.9 -> 64.8; 2.9% | Helped |
| Use DeepSeek-R1 rather than DeepSeek-V3 as the pairwise evaluator | SciArena-Eval pairwise evaluation · SciArena-Eval | accuracy: 60.5 -> 61.2; 0.7% | Helped |
| Include supporting citations | Responses evaluated in SciArena · SciArena | Bradley-Terry coefficient: γ = 0.155 | Helped |
| Include irrelevant or contradicting citations | Responses evaluated in SciArena · SciArena | Bradley-Terry coefficient: γ = -0.154 | Hurt |
Surprising findings
- Citation count has only a modest preference effect (0.039) compared with Search Arena (0.209), while supporting versus irrelevant citation attribution matters strongly.
- Open-source models such as GPT-OSS-120B and DeepSeek-R1-0528 outrank several proprietary models such as o4-mini and Claude-4-Sonnet.
Limitations
- Excludes older or deprecated models such as GPT-4o, Llama-3.1 and Qwen-2.5, and cannot integrate Deep Research agents due to usage caps and lack of APIs.
- Community vote quality checks rely on small manual samples (133 votes) and an IP-based anomaly detector.
- Evaluator accuracy on SciArena-Eval remains low (best 65.1%), so automated judges are not yet reliable substitutes for experts.
Experimental setup
Datasets: SciArena human votes (20,832 votes)
Benchmarks: SciArena-Eval SciArena leaderboard
Metrics: Elo score (Bradley-Terry) accuracy vs human votes inter-annotator agreement self-consistency Bradley-Terry style coefficients
Baselines: Chatbot Arena Search Arena AlpacaEval WildChat
Builds on: Chatbot Arena ScholarQA Bradley-Terry model
Closest papers by meaning
- SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research (AAAI 2024 · similarity 0.88)
- Prometheus: Inducing Fine-Grained Evaluation Capability in Language Models (ICLR 2024 · similarity 0.87)
- SciCode: A Research Coding Benchmark Curated by Scientists (NeurIPS 2024 · similarity 0.87)
- CycleResearcher: Improving Automated Research via Automated Review (ICLR 2025 · similarity 0.87)
- PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistencies (ICLR 2026 · similarity 0.87)
- AAAR-1.0: Assessing AI’s Potential to Assist Research (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.