Research map / Large language models / Agents, code and math
Scaling Generative Verifiers For Natural Language Mathematical Proof Verification And Selection
Sadegh Mahdavi, Branislav Kisacanin, Shubham Toshniwal, Wei Du, Ivan Moshkov, George Armstrong, Renjie Liao, Christos Thrampoulidis, Igor Gitman
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| a simple MLP on text embeddings, with no math checking, exploits dataset imbalance | VerOPC test set · VerOPC | accuracy (%): 75.34% accuracy, more than 5% above Claude-Sonnet-4 | Improved |
| use RL (GRPO) to train the judge (ablation) | Qwen3-30B-A3B-Thinking-2507 LLM-as-a-Judge · VerProofArena, OPC prompt | F1: 37.23 -> 68.75 | Improved |
| add RL training to the judge (ablation) | LLM-as-a-Judge prompts · VerProofArena | prompt sensitivity (F1 gap across prompts): OPC 68.75 vs GIMO 65.40 after RL | Improved |
| add RL training to the judge (ablation) | OPC-prompt judge · Challenge-19 final-answer precision | precision: 60.41 -> 62.59 | No clear effect |
| add rubric at test time to a model trained without rubric (ablation) | OPC RL judge · VerProofArena and Challenge-19 | F1 and final-answer precision: F1 68.75 -> 73.35; precision 62.59 -> 99.31 | Improved |
| add rubric or ground-truth proof to the judge prompt in both training and evaluation (ablation) | OPC RL judge · VerProofArena | F1: 68.75 -> 56.69 | Hurt |
| add rubric or ground-truth proof to the prompt (ablation) | LLM-as-a-Judge and GenSelect · SelProofBench, SelOPC proof selection | selection accuracy | No clear effect |
| replace binary grading prompt with 7-point grading (ablation) | LLM-as-a-Judge · SelProofBench, SelOPC | selection accuracy | No clear effect |
| ensemble different judges by averaging scores (ablation) | LLM-as-a-Judge · SelOPC | selection accuracy | No clear effect |
| use step-based judgement with all-steps-correct aggregation (ablation) | LLM-as-a-Judge · proof verification | accuracy and recall | Hurt |
| increase number of sampled judgements per proof up to 32 (ablation) | LLM-as-a-Judge · proof selection | selection accuracy | Improved |
| use hybrid knockout GenSelect plus LLM-as-a-Judge instead of majority voting | GPT-OSS-120B · Challenge-19 | accuracy (%): 82.23 -> 96.05 | Improved |
Closest papers by meaning
- Variation in Verification: Understanding Verification Dynamics in Large Language Models (ICLR 2026 · similarity 0.92)
- Incentivizing LLMs to Self-Verify Their Answers (NeurIPS 2025 · similarity 0.91)
- Generative Verifiers: Reward Modeling as Next-Token Prediction (ICLR 2025 · similarity 0.90)
- Reinforcing General Reasoning Without Verifiers (ICLR 2026 · similarity 0.90)
- T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling (ICML 2025 · similarity 0.90)
- Trust, But Verify: A Self-Verification Approach to Reinforcement Learning with Verifiable Rewards (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
