Research map / Large language models / Agents, code and math
Judging What We Cannot Solve: A Consequence-Based Approach for Oracle-Free Evaluation of Research-Level Math
Guijin Son, Donghun Yang, Hitesh Laxmichand Patel, Hyunwoo Ko, Amit Agarwal, Sunghee Ahn, Kyong-Ha Lee, Youngjae Yu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace LLM-judge scoring with Consequence-Based Utility | GPT-OSS-120B · ExpertMath ranking | Acc@1: 67.21 -> 76.27 | Improved |
| replace LLM-judge scoring with Consequence-Based Utility | GPT-OSS-120B · ExpertMath ranking | HumanWin: 48.57 -> 82.86 | Improved |
| replace LLM-judge scoring with Consequence-Based Utility | Qwen3-235B-A22B · ExpertMath ranking | Recall@5: 80.02 -> 74.15 | Hurt |
| replace LLM-judge scoring with CBU | GPT-OSS-20B · ExpertMath ranking | AUC: 69.03 -> 79.18 | Improved |
| use CBU score instead of LLM-judge score as logistic regression feature (ablation) | Qwen3-30B-A3B · ExpertMath correctness prediction | accuracy: 58.06 -> 76.31 | Improved |
| combine CBU and LLM-judge scores as features (ablation) | Qwen3-235B-A22B · ExpertMath correctness prediction | accuracy: 72.69 -> 79.65 | Improved |
| combine CBU and LLM-judge scores as features (ablation) | GPT-OSS-20B · ExpertMath correctness prediction | accuracy: 73.09 -> 73.90 | No clear effect |
| use CBU instead of LLM-judge on hardest (unsolved) questions | GPT-OSS-120B · ExpertMath, score gap vs difficulty | correct-wrong score gap | Improved |
| use CBU with LLM-generated neighborhoods instead of LLM-judge | GPT-OSS-20B · RealMath (2) | AUC: 48.76 -> 69.83 | Improved |
| use CBU instead of LLM-judge on easy contest-level variants | GPT-OSS-20B · Daft-Math | Acc@1: 93.51 -> 85.58 | Hurt |
| increase number of rollouts from 4/8 to 64 (ablation) | CBU · ExpertMath utility estimate | normalized error to 64-rollout reference | Improved |
| use reward models with a single scoring pass instead of 64 rollouts | AceMath-72B-RM · ExpertMath ranking | AUC: 20.75 AUC | Hurt |
Closest papers by meaning
- The Quality-Utility Paradox: Why High-Reward Data Impairs Small Model Mathematical Reasoning (ICML 2026 · similarity 0.89)
- Escaping the Cognitive Well: Efficient Competition Math with Off-the-Shelf Models (NeurIPS 2026 · similarity 0.89)
- RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics (NeurIPS 2025 · similarity 0.87)
- Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation (ICML 2025 · similarity 0.87)
- Not All Proofs Are Equal: Evaluating LLM Proof Quality Beyond Correctness (NeurIPS 2026 · similarity 0.87)
- MARGE: Improving Math Reasoning with Guided Exploration (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
