Research map / Large language models / Agents, code and math
The Open Proof Corpus: A Large-Scale Study of LLM-Generated Mathematical Proofs
Jasper Dekoninck, Ivo Petrov, Kristian Minchev, Miroslav Marinov, Maria Drencheva, Lyuba Konova, Milen Milenov Shumanov, Kaloyan Tsvetkov, Nikolay Drenchev, Lazar D. Todorov, Kalina Nikolova, Nikolay Georgiev and 4 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| GRPO fine-tune R1-Qwen3-8B on OPC human labels | R1-Qwen3-8B · OPC proof judging | accuracy: 70.7 -> 83.8 (pass@1); 71.3 -> 88.1 (maj@5) | Improved |
| fine-tune on OPC (high-school only), test on undergraduate problems | R1-Qwen3-8B · Putnam undergraduate judging (OOD) | accuracy: 72.1 -> 75.0 | Improved |
| judge own proofs instead of others' proofs | LLM judges · OPC judging breakdown | accuracy: e.g. Gemini 79.4 own vs 86.9-85.9 by others | Hurt |
| provide ground-truth solution to the judge (ablation) | LLM judges · OPC proof judging | accuracy: GPT-5 89.3 -> 89.0; o3 83.1 -> 85.9; Gemini-2.5-Pro 85.4 -> 82.7 | No clear effect |
| require valid proof instead of correct final answer | o3 · MathArena subset | accuracy: 59.5% of answers contain correct proof | Hurt |
| use best-of-n selection with o4-mini judge | o4-mini · OPC best-of-n subset | proof accuracy: 26% -> 36% (standard), 47% (ranking) | Improved |
| use pairwise ranking selection instead of discrete or continuous scoring | best-of-n · OPC best-of-n small subset | proof accuracy: about +10% | Improved |
| use Rank (Swiss) instead of Rank (Bracket) | best-of-n · OPC best-of-n small subset | proof accuracy: +3% margin | Improved |
| increase n beyond 5 for discrete and continuous selection | best-of-n · OPC best-of-n small subset | proof accuracy | No clear effect |
| natural-language instead of formal proof generation | Gemini-2.5-Pro vs Goedel-Prover-v2 · PutnamBench subset | proof accuracy: <19% -> almost 83% | Improved |
Closest papers by meaning
- Not All Proofs Are Equal: Evaluating LLM Proof Quality Beyond Correctness (NeurIPS 2026 · similarity 0.94)
- FaithSieve: Fine-Grained Evaluation of Math Proofs with Faithful Formal Evidence (NeurIPS 2026 · similarity 0.93)
- One Example Shown, Many Concepts Known! Counterexample-Driven Conceptual Reasoning in Mathematical LLMs (ICML 2025 · similarity 0.92)
- MathGAP: Out-of-Distribution Evaluation on Problems with Arbitrarily Complex Proofs (ICLR 2025 · similarity 0.91)
- Llemma: An Open Language Model for Mathematics (ICLR 2024 · similarity 0.91)
- Solving Inequality Proofs with Large Language Models (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
