AI Paper Atlas

Research map / Large language models / Agents, code and math

Reliable Fine-Grained Evaluation of Natural Language Math Proofs

Wenjie Ma, Andrei Cojocaru, Neel Kolhe, Haihan Zhang, Vincent Zhuang, Matei Zaharia, Sewon Min

ICLR 2026 · Large language models › Agents, code and math · approach: Mathematical and structural reasoning benchmarks

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add problem-specific marking scheme and reference solution to evaluator context (ablation)o3 evaluator · ProofBenchMAE: 1.680 -> 0.964Improved
add marking scheme only (without reference solution) (ablation)o3 evaluator · ProofBenchMAE: 1.680 -> 1.069Improved
add reference solution only (ablation)o3 evaluator · ProofBenchMAE: 1.680 -> 1.330Improved
add reference solution to marking scheme for weaker backbones (ablation)Gemini / gpt-5 / o4-mini / R1 evaluators · ProofBenchMAE: Gemini 1.142 (MS) vs 1.342 (Ref+MS)No clear effect
use a stronger reasoning backbone (o3 vs gpt-4o) (ablation)evaluator with Ref+MS · ProofBenchMAE: 2.197 -> 0.964Improved
use strict instruction instead of flexible Norm (ablation)o3 evaluator · ProofBench Ref+MSMAE: 0.964 -> 1.095Hurt
use strict instruction instead of Norm (ablation)Gemini and o4-mini evaluators · ProofBench Ref+MSMAE: Gemini 1.342 -> 1.231; o4-mini 1.367 -> 1.266Improved
ensemble five o3 runs (median aggregation) (ablation)o3 evaluator · ProofBench Ref+MSMAE: 0.944 (best single) -> 0.926Improved
ensemble by mean over five runs (ablation)o3 evaluator · ProofBench Ref+MSRMSE: 1.225 -> 1.169Improved
add two-step Binary+Errors then Fine-Grained pipeline (ablation)o4-mini evaluator · ProofBench Ref+MSMAE: 1.367 -> 1.245Improved
add two-step Binary+Errors then Fine-Grained pipeline (ablation)o3 evaluator · ProofBench Ref+MSMAE: 0.964 -> 1.065Hurt
add self-reflection and final verdict stages (ablation)o3 and o4-mini evaluators · ProofBenchMAE: o3 1.107 -> 1.055 (reflection), 1.058 (verdict)No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy