AI Paper Atlas

Research map / Large language models / Agents, code and math

Can LLMs Reliably Grade Olympiad Proofs? A Controlled Study of Mathematical Verification with LLMs

Azim Ospanov, Zijin Feng, Ding Ding, Chengwu Liu, Haoli Bai, Jiacheng Sun, Lifeng Shang, Farzan Farnia

NeurIPS 2026 · Large language models › Agents, code and math · approach: Formal verification and proof data

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

A controlled study of whether LLMs can reliably verify and grade natural-language Olympiad-level mathematical proofs.

Problem

Verifying natural-language mathematical proofs relies on costly expert inspection, and Olympiad-level proofs demand meticulous step-by-step evaluation that LLMs may not handle reliably.

Approach

Systematic, controlled evaluation of LLMs as graders of Olympiad-level proofs; the abstract does not specify the method beyond this framing.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace MajorityVote@16 with MABGrader (UCB over grades)Qwen3.5-27B · Trojan-Bench(H) FullQWK: 0.41 -> 0.57Improved
replace ProofGrader MajorityVote@16 with MABGraderGemini-3-Pro-Preview · Trojan-Bench(H) FullQWK: 0.46 -> 0.63Improved
replace ProofGrader MajorityVote@16 with MABGraderKimi-K2.5 · Trojan-Bench(H) FullRMSE: 2.84 -> 2.65Improved
replace ProofGrader with MABGraderDeepSeek-V4-Flash · Trojan-Bench(H) FullQWK: 0.66 -> 0.66No clear effect
replace reward-model Best-of-N with MABGraderDeepSeek-V4-Flash · Trojan-Bench(H) FullRMSE and Bias: RMSE 2.02 (RLHFlow) vs 2.06Hurt
use Best-of-N selection with ORM/PRM reward modelsKimi-K2.5, Qwen3.5-27B · Trojan-Bench(H) FullQWK: 0.20 -> 0.15 (ArmoRM, Kimi-K2.5)No clear effect
fine-tune on synthetic Trojan-Bench(S) (ablation)Qwen3.5-9B · Trojan-Bench(S) FullQWK: 0.02 -> 0.35Improved
fine-tune on synthetic Trojan-Bench(S) (ablation)Qwen3.5-27B (LoRA rank 8) · Trojan-Bench(S) FullQWK: 0.36 -> 0.47Improved
fine-tune on synthetic Trojan-Bench(S) (ablation)Qwen3.5-9B · Trojan-Bench(S) BlindQWK: 0.06 -> 0.07No clear effect
provide reference solution and marking scheme (Full vs Blind) (ablation)LLM verifiers · Trojan-Bench(H)QWK: +0.06 absolute correlationImproved
use MajorityVote@16 instead of single CoT@1 (ablation)LLM verifiers · Trojan-Bench(H) FullQWK: +5.46% vs CoTImproved
use a thinking variant of the model (ablation)Claude-Sonnet-4.6 · Trojan-Bench(H) FullQWK: 0.446 -> 0.513Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy