AI Paper Atlas

Research map / Large language models / Agents, code and math

Judging What We Cannot Solve: A Consequence-Based Approach for Oracle-Free Evaluation of Research-Level Math

Guijin Son, Donghun Yang, Hitesh Laxmichand Patel, Hyunwoo Ko, Amit Agarwal, Sunghee Ahn, Kyong-Ha Lee, Youngjae Yu

ICML 2026 · Large language models › Agents, code and math · approach: Inductive and abstract reasoning evaluation · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace LLM-judge scoring with Consequence-Based UtilityGPT-OSS-120B · ExpertMath rankingAcc@1: 67.21 -> 76.27Improved
replace LLM-judge scoring with Consequence-Based UtilityGPT-OSS-120B · ExpertMath rankingHumanWin: 48.57 -> 82.86Improved
replace LLM-judge scoring with Consequence-Based UtilityQwen3-235B-A22B · ExpertMath rankingRecall@5: 80.02 -> 74.15Hurt
replace LLM-judge scoring with CBUGPT-OSS-20B · ExpertMath rankingAUC: 69.03 -> 79.18Improved
use CBU score instead of LLM-judge score as logistic regression feature (ablation)Qwen3-30B-A3B · ExpertMath correctness predictionaccuracy: 58.06 -> 76.31Improved
combine CBU and LLM-judge scores as features (ablation)Qwen3-235B-A22B · ExpertMath correctness predictionaccuracy: 72.69 -> 79.65Improved
combine CBU and LLM-judge scores as features (ablation)GPT-OSS-20B · ExpertMath correctness predictionaccuracy: 73.09 -> 73.90No clear effect
use CBU instead of LLM-judge on hardest (unsolved) questionsGPT-OSS-120B · ExpertMath, score gap vs difficultycorrect-wrong score gapImproved
use CBU with LLM-generated neighborhoods instead of LLM-judgeGPT-OSS-20B · RealMath (2)AUC: 48.76 -> 69.83Improved
use CBU instead of LLM-judge on easy contest-level variantsGPT-OSS-20B · Daft-MathAcc@1: 93.51 -> 85.58Hurt
increase number of rollouts from 4/8 to 64 (ablation)CBU · ExpertMath utility estimatenormalized error to 64-rollout referenceImproved
use reward models with a single scoring pass instead of 64 rolloutsAceMath-72B-RM · ExpertMath rankingAUC: 20.75 AUCHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy