Research map / Large language models / Reasoning and chain of thought

An Enigma of Artificial Reason: Investigating the Production-Evaluation Gap in Large Reasoning Models

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Long CoT dynamics and efficiency · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces the VAIR dataset of math solutions with trivial reasoning flaws but valid answers to show that large reasoning models fail to evaluate reasoning even though they produce it near-perfectly, and traces this to answer confirmation bias via CoT analysis, linear probes, and causal patching.

Problem

Large reasoning models excel at producing reasoning but it is unclear how well they evaluate reasoning when the final answer is valid; existing evaluations confound reasoning evaluation with answer correctness.

Approach

Decouple reasoning evaluation from production and answer validity using VAIR (valid final answer, invalid reasoning) solutions, then diagnose the failure with CoT annotation, dynamic linear probes, and causal activation patching of answer tokens, revealing an answer confirmation bias.

Key results

• Frontier LRMs score as low as 47.9% (GPT 5.4) and 52.5% (GPT 5) on VAIR evaluation while exceeding 94.7% on solving; Gemini 3.1 Pro reaches 78.6%.
• Human participants (n=195) show only a maximum 6.3% gap, with 74.5% VAIR accuracy vs 80.8% on solving.
• Causal patching of answer activations flips verdicts at rates of 80.5% (Qwen3-0.6B), 52.2% (Qwen3-4B), and 55.6% (GPT-oss-20B) when patching all layers.

Surprising findings

• Humans outperform most LRMs on VAIR evaluation in absolute terms despite lower solving accuracy than frontier models.
• Process reward model Qwen2.5-Math-PRM-7B shows similar VAIR failures (67.8% overall, 49.1% on Missing Reasoning).

Limitations

• Mechanistic analyses restricted to open-weight models under 20B parameters.
• Evaluation limited to mathematical reasoning where validity is clearly defined.
• Does not directly test the effect of LRM training objectives on the gap.

Experimental setup

Datasets: GSM8K MATH Process-Bench VAIR VAVR IAIR

Benchmarks: VAIR VAVR IAIR

Metrics: accuracy flip rate probe accuracy P(model represents solution as valid)

Baselines: Claude Sonnet 4.6 Claude Opus 4.7 DeepSeek R1 GPT 5 GPT 5.4 Gemini 3.1 Pro Qwen2.5-Math-PRM-7B

Builds on: GSM8K MATH Process-Bench process reward models

Closest papers by meaning

• The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity (NeurIPS 2025 · similarity 0.92)
• Measuring the Faithfulness of Thinking Drafts in Large Reasoning Models (NeurIPS 2025 · similarity 0.90)
• Decoding the Critique Mechanism in Large Reasoning Models (NeurIPS 2026 · similarity 0.90)
• Answering the Unanswerable Is to Err Knowingly: Analyzing and Mitigating Abstention Failures in Large Reasoning Models (AAAI 2026 · similarity 0.89)
• RFEval: Benchmarking Reasoning Faithfulness under Counterfactual Reasoning Intervention in Large Reasoning Models (ICLR 2026 · similarity 0.89)
• On Reasoning Strength Planning in Large Reasoning Models (NeurIPS 2025 · similarity 0.89)