Research map / Large language models / Reasoning and chain of thought
Escaping the Cognitive Well: Efficient Competition Math with Off-the-Shelf Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a token-efficient inference pipeline (Momus) using off-the-shelf Gemini models that uses conjecture extraction and contextual detachment to escape "Cognitive Well" traps in solver-grader loops, reaching 87.6% on IMO-ProofBench Advanced at about $9 per question.
Problem
Solver-grader LLM pipelines for olympiad proof generation rely on massive parallelism and thousands of calls, yet their internal graders can be fooled into accepting flawed proofs (the "Cognitive Well"), making them very expensive.
Approach
Limit parallelism to K=4 solvers with adversarial grading, extract load-bearing conjectures from stalled proofs, prove each conjecture and its negation in a fresh context (contextual detachment), and feed proven lemmas into a global memory for subsequent solver rounds.
Key results
- On IMO-ProofBench Advanced, the pipeline with Gemini 3.1 Pro reaches 87.6% at about $9 per question, per the abstract.
- With Gemini 2.5 Pro, a token-matched no-conjecture ablation scores 34.3% versus 55% for the full pipeline on PB-Advanced.
- Using Claude 4.6 Opus as backbone, the pipeline rises from a 23.8% base call to 85.7% on PB-Advanced; DeepSeek-V3.2 rises from 15.2% to 66.7%.
Surprising findings
- The grader accepted a flawed solution (the solver explicitly stated it had not finished) 6 of 20 times, and 0 of 20 when the admission was truncated, suggesting grader reward hacking.
- Momus grader has lower accuracy on GraderBench than several baselines yet performs better inside the pipeline, indicating standalone grader accuracy does not align with pipeline utility.
Limitations
- Evaluation on only 30 PB-Advanced problems, with run-to-run noise; no strong monotonic trend in hyperparameter sweeps.
- Headline numbers rely on closed Gemini models; many comparisons use results taken from other papers and autograder rather than full re-runs.
- Expert grading covers only a single run each for Gemini 2.5 Pro and 3 Pro.
Experimental setup
Datasets: IMO-ProofBench Advanced IMO-ProofBench Basic GraderBench ProofBench IMO-AnswerBench
Benchmarks: IMO-ProofBench Advanced IMO-ProofBench Basic GraderBench ProofBench IMO-AnswerBench
Metrics: autograded score (0-7 average %) expert human grade accuracy MAE FPR FNR cost per question
Baselines: DeepSeekMath V2 Gemini Deep Think (IMO Gold) Aletheia Huang-Yang pipeline QED-Nano ProofAutoGrader
Builds on: PDR (parallel-distill-refine) DeepSeekMath V2 Huang-Yang solver prompt
Closest papers by meaning
- Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation (ICML 2025 · similarity 0.88)
- MathSmith: Towards Extremely Hard Mathematical Reasoning by Forging Synthetic Problems with a Reinforced Policy (AAAI 2026 · similarity 0.88)
- MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning (ICLR 2024 · similarity 0.88)
- UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models (ICLR 2025 · similarity 0.88)
- RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics (NeurIPS 2025 · similarity 0.88)
- A Careful Examination of Large Language Model Performance on Grade School Arithmetic (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.