Research map / Large language models / Agents, code and math
QEDBench: Quantifying the Alignment Gap in Automated Evaluation of University-Level Mathematical Proofs
Santiago Gonzalez, Alireza Amiri Bavandpour, Peter Ye, Edward Zhang, Ruslans Aleksejevs, Todor Antić, Polina Baron, Sujeet Bhalerao, Shubhrajit Bhattacharya, Zachary Burton, John Byrne, Hyungjun Choi and 39 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add course-specific rubric penalizing out-of-scope machinery to the judge prompt (ablation) | GPT-5.2 Pro judge · QEDBench alignment with human consensus | Pearson r: 0.69 -> 0.67 | No clear effect |
| add course-specific rubric penalizing out-of-scope machinery to the judge prompt (ablation) | GPT-5.2 Pro judge · QEDBench alignment with human consensus | MAE: 0.13 -> 0.14 | No clear effect |
| switch from expert to course-specific rubric for LLM judging of solvers (ablation) | Gemini 3.0 Pro solutions · QEDBench | average score: 0.957 -> 0.926 | Hurt |
| switch from expert to course-specific rubric for LLM judging of solvers (ablation) | Claude Sonnet 4.5 solutions · QEDBench | average score: 0.883 -> 0.866 | Hurt |
| solving constructive combinatorics problems versus procedural ODE problems | Claude Sonnet 4.5 · QEDBench pass rate | pass rate: 100.0% (ODEs) -> 27.3% (Combinatorics) | Hurt |
| compare online-available problems with non-online ones (contamination) | human-evaluated solver performance · QEDBench 272 problems | mean score / pass rate gap: +0.017 mean-score gap, +0.011 pass-rate gap | No clear effect |
| control for human proof quality and judge strictness in same-family self-preference regression | LLM judges · QEDBench | self-preference effect: -0.014 to +0.033 | No clear effect |
| use strict binary Pass/Fail judge prompts instead of scores (ablation) | LLM judges · QEDBench | leniency trend | No clear effect |
Closest papers by meaning
- Not All Proofs Are Equal: Evaluating LLM Proof Quality Beyond Correctness (NeurIPS 2026 · similarity 0.90)
- RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics (NeurIPS 2025 · similarity 0.90)
- FaithSieve: Fine-Grained Evaluation of Math Proofs with Faithful Formal Evidence (NeurIPS 2026 · similarity 0.89)
- UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models (ICLR 2025 · similarity 0.89)
- Putnam-AXIOM: A Functional & Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs (ICML 2025 · similarity 0.89)
- Solving Inequality Proofs with Large Language Models (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
