Research map / Large language models / Agents, code and math
Not All Proofs Are Equal: Evaluating LLM Proof Quality Beyond Correctness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ProofRank, a benchmark of 382 final-answer competition problems that scores LLM-written proofs on scalable proxies of proof quality (conciseness, computational ease, cognitive simplicity, diversity, adaptivity) via pairwise comparisons, and evaluates 10 frontier models.
Problem
Existing proof benchmarks evaluate only correctness, treating all correct proofs as equally good, although proofs differ in clarity, conciseness, insight, and transferability.
Approach
Measure measurable proxies of proof quality (compressibility, computational ease, cognitive simplicity, technique diversity, technique adaptivity) using pairwise comparisons among correct solutions, aggregated with Bradley-Terry ratings, instead of absolute scores.
Key results
- GPT-5.4 achieves 86.4% accuracy and the top rating on computational ease (1615), cognitive simplicity (1330) and adaptivity (1558).
- Gemini-3.1-Pro is extremely verbose, with solutions on average 3.5 times more compressible than necessary (conciseness Elo 272 vs 1535 for Kimi-K2.5-Think).
- Spearman correlation between accuracy and quality metrics is low; computational ease and conciseness correlate near 0 with accuracy, while adaptivity correlates 0.84.
Surprising findings
- Including incorrect solutions substantially changes rankings; Step-3.5-Flash's ratings jump because its incorrect solutions look simple.
- Prompting for conciseness cuts accuracy by about 15% for Kimi-K2.5-Think and DeepSeek-V3.2.
Limitations
- Proof quality is not directly measured; proxies are scalable but imperfect.
- LLM judges may be biased or misaligned with human judgments, with conciseness showing low human agreement (57.6%).
- Restricted to final-answer high-school-level competition problems, limiting generality to research-level proofs.
Experimental setup
Datasets: MathArena IMO-AnswerBench human AoPS solutions
Benchmarks: ProofRank
Metrics: Elo-style Bradley-Terry rating accuracy compressibility ratio Spearman correlation Rand Index inter-annotator agreement
Baselines: GPT-5.4 Gemini-3.1-Pro Kimi-K2.5-Think GLM-5 DeepSeek-V3.2 Step-3.5-Flash GPT-OSS-120B Grok-4.1-Fast Gemini-3-Flash Qwen3.5-397B-A17B
Builds on: Bradley-Terry model MathArena IMO-AnswerBench
Closest papers by meaning
- RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics (NeurIPS 2025 · similarity 0.92)
- FaithSieve: Fine-Grained Evaluation of Math Proofs with Faithful Formal Evidence (NeurIPS 2026 · similarity 0.92)
- MathArena: Evaluating LLMs on Uncontaminated Math Competitions (NeurIPS 2025 · similarity 0.92)
- MathConstruct: Challenging LLM Reasoning with Constructive Proofs (ICML 2025 · similarity 0.91)
- Solving Inequality Proofs with Large Language Models (NeurIPS 2025 · similarity 0.91)
- One Example Shown, Many Concepts Known! Counterexample-Driven Conceptual Reasoning in Mathematical LLMs (ICML 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.