Research map / Large language models / Agents, code and math
Can LLMs Reliably Grade Olympiad Proofs? A Controlled Study of Mathematical Verification with LLMs
Azim Ospanov, Zijin Feng, Ding Ding, Chengwu Liu, Haoli Bai, Jiacheng Sun, Lifeng Shang, Farzan Farnia
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
A controlled study of whether LLMs can reliably verify and grade natural-language Olympiad-level mathematical proofs.
Problem
Verifying natural-language mathematical proofs relies on costly expert inspection, and Olympiad-level proofs demand meticulous step-by-step evaluation that LLMs may not handle reliably.
Approach
Systematic, controlled evaluation of LLMs as graders of Olympiad-level proofs; the abstract does not specify the method beyond this framing.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace MajorityVote@16 with MABGrader (UCB over grades) | Qwen3.5-27B · Trojan-Bench(H) Full | QWK: 0.41 -> 0.57 | Improved |
| replace ProofGrader MajorityVote@16 with MABGrader | Gemini-3-Pro-Preview · Trojan-Bench(H) Full | QWK: 0.46 -> 0.63 | Improved |
| replace ProofGrader MajorityVote@16 with MABGrader | Kimi-K2.5 · Trojan-Bench(H) Full | RMSE: 2.84 -> 2.65 | Improved |
| replace ProofGrader with MABGrader | DeepSeek-V4-Flash · Trojan-Bench(H) Full | QWK: 0.66 -> 0.66 | No clear effect |
| replace reward-model Best-of-N with MABGrader | DeepSeek-V4-Flash · Trojan-Bench(H) Full | RMSE and Bias: RMSE 2.02 (RLHFlow) vs 2.06 | Hurt |
| use Best-of-N selection with ORM/PRM reward models | Kimi-K2.5, Qwen3.5-27B · Trojan-Bench(H) Full | QWK: 0.20 -> 0.15 (ArmoRM, Kimi-K2.5) | No clear effect |
| fine-tune on synthetic Trojan-Bench(S) (ablation) | Qwen3.5-9B · Trojan-Bench(S) Full | QWK: 0.02 -> 0.35 | Improved |
| fine-tune on synthetic Trojan-Bench(S) (ablation) | Qwen3.5-27B (LoRA rank 8) · Trojan-Bench(S) Full | QWK: 0.36 -> 0.47 | Improved |
| fine-tune on synthetic Trojan-Bench(S) (ablation) | Qwen3.5-9B · Trojan-Bench(S) Blind | QWK: 0.06 -> 0.07 | No clear effect |
| provide reference solution and marking scheme (Full vs Blind) (ablation) | LLM verifiers · Trojan-Bench(H) | QWK: +0.06 absolute correlation | Improved |
| use MajorityVote@16 instead of single CoT@1 (ablation) | LLM verifiers · Trojan-Bench(H) Full | QWK: +5.46% vs CoT | Improved |
| use a thinking variant of the model (ablation) | Claude-Sonnet-4.6 · Trojan-Bench(H) Full | QWK: 0.446 -> 0.513 | Improved |
Closest papers by meaning
- Proving Olympiad Inequalities by Synergizing LLMs and Symbolic Reasoning (ICLR 2025 · similarity 0.92)
- MathArena: Evaluating LLMs on Uncontaminated Math Competitions (NeurIPS 2025 · similarity 0.91)
- Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation (ICML 2025 · similarity 0.91)
- Not All Proofs Are Equal: Evaluating LLM Proof Quality Beyond Correctness (NeurIPS 2026 · similarity 0.90)
- LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming? (NeurIPS 2025 · similarity 0.89)
- One Example Shown, Many Concepts Known! Counterexample-Driven Conceptual Reasoning in Mathematical LLMs (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
