Research map / Large language models / Agents, code and math
Reliable Fine-Grained Evaluation of Natural Language Math Proofs
Wenjie Ma, Andrei Cojocaru, Neel Kolhe, Haihan Zhang, Vincent Zhuang, Matei Zaharia, Sewon Min
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add problem-specific marking scheme and reference solution to evaluator context (ablation) | o3 evaluator · ProofBench | MAE: 1.680 -> 0.964 | Improved |
| add marking scheme only (without reference solution) (ablation) | o3 evaluator · ProofBench | MAE: 1.680 -> 1.069 | Improved |
| add reference solution only (ablation) | o3 evaluator · ProofBench | MAE: 1.680 -> 1.330 | Improved |
| add reference solution to marking scheme for weaker backbones (ablation) | Gemini / gpt-5 / o4-mini / R1 evaluators · ProofBench | MAE: Gemini 1.142 (MS) vs 1.342 (Ref+MS) | No clear effect |
| use a stronger reasoning backbone (o3 vs gpt-4o) (ablation) | evaluator with Ref+MS · ProofBench | MAE: 2.197 -> 0.964 | Improved |
| use strict instruction instead of flexible Norm (ablation) | o3 evaluator · ProofBench Ref+MS | MAE: 0.964 -> 1.095 | Hurt |
| use strict instruction instead of Norm (ablation) | Gemini and o4-mini evaluators · ProofBench Ref+MS | MAE: Gemini 1.342 -> 1.231; o4-mini 1.367 -> 1.266 | Improved |
| ensemble five o3 runs (median aggregation) (ablation) | o3 evaluator · ProofBench Ref+MS | MAE: 0.944 (best single) -> 0.926 | Improved |
| ensemble by mean over five runs (ablation) | o3 evaluator · ProofBench Ref+MS | RMSE: 1.225 -> 1.169 | Improved |
| add two-step Binary+Errors then Fine-Grained pipeline (ablation) | o4-mini evaluator · ProofBench Ref+MS | MAE: 1.367 -> 1.245 | Improved |
| add two-step Binary+Errors then Fine-Grained pipeline (ablation) | o3 evaluator · ProofBench Ref+MS | MAE: 0.964 -> 1.065 | Hurt |
| add self-reflection and final verdict stages (ablation) | o3 and o4-mini evaluators · ProofBench | MAE: o3 1.107 -> 1.055 (reflection), 1.058 (verdict) | No clear effect |
Closest papers by meaning
- FaithSieve: Fine-Grained Evaluation of Math Proofs with Faithful Formal Evidence (NeurIPS 2026 · similarity 0.94)
- Not All Proofs Are Equal: Evaluating LLM Proof Quality Beyond Correctness (NeurIPS 2026 · similarity 0.93)
- MUSTARD: Mastering Uniform Synthesis of Theorem and Proof Data (ICLR 2024 · similarity 0.91)
- MathGAP: Out-of-Distribution Evaluation on Problems with Arbitrarily Complex Proofs (ICLR 2025 · similarity 0.91)
- RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics (NeurIPS 2025 · similarity 0.91)
- MathArena: Evaluating LLMs on Uncontaminated Math Competitions (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
