Research map / Large language models / Reasoning and chain of thought

Verifying Meta-Awareness via Predictive Rewards in Reasoning Models

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MAPR, an RL post-training method where the policy also predicts its own rollout statistics (pass-rate, correct-response length, useful notions) and is rewarded for accuracy of those predictions, plus a faster variant using predictive gating and early cutoff.

Problem

Reasoning models with GRPO are trained using only answer-based verification, ignoring whether they know their own difficulty, length budget and knowledge boundaries; this limits training efficiency and accuracy on math reasoning.

Approach

A parallel meta-prompt makes the policy predict pass-rate, correct-solution length and problem-solving notions, and rewards these predictions against statistics computed from the policy's own solution rollouts (self-generated dynamic ground truth). The predictions are then used for predictive gating (skipping trivial/unsolvable prompts before rollout), early length cutoff, and notion hints.

Key results

• On Qwen3-4B, MAPR raises average Pass@1 over six math benchmarks from 42.49 (GRPO) to 49.48 (+13.04%), with AIME25 gaining 83.18% relative.
• MAPR speeds up GRPO training by over 1.28x to reach the same performance, per the abstract.
• On Qwen3-8B with DAPO, DAPO + MAPR improves average Pass@1 from 53.33 to 55.01 and Pass@8 from 69.23 to 73.49.

Surprising findings

• Adding notion feed-in to MAPR gives only small gains on AIME25 and sometimes decreases performance, which the authors attribute to information overlap with what the model already implicitly knows.
• Meta-prediction rollouts add only about 15.5% of total generated tokens, so the self-prediction overhead is modest.

Limitations

• Gains on some benchmarks are mixed, e.g., Minerva Pass@1 drops for Qwen3-4B and AMC23 Pass@8 drops for Qwen3-14B.
• Predictive gating is static online, so gated prompts are never re-evaluated.
• Llama and Gemma experiments use the easier MATH train split and exclude AIME due to near-zero accuracy.

Experimental setup

Datasets: DeepScaleR MATH train split

Benchmarks: AIME24 AIME25 AMC23 MATH500 Minerva OlympiadBench ProntoQA ProofWriter FOLIO LogicalDeduction AR-LSAT GPQA Diamond

Metrics: Pass@1 Pass@8

Baselines: GRPO DAPO GRPO + NF

Builds on: GRPO DAPO VeRL

Closest papers by meaning

• MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation (ICLR 2025 · similarity 0.89)
• On the self-verification limitations of large language models on reasoning and planning tasks (ICLR 2025 · similarity 0.88)
• LIMOPro: Reasoning Refinement for Efficient and Effective Test-time Scaling (NeurIPS 2025 · similarity 0.87)
• MR-Ben: A Meta-Reasoning Benchmark for Evaluating System-2 Thinking in LLMs (NeurIPS 2024 · similarity 0.87)
• LYNX: Learning Dynamic Exits for Confidence-Controlled Reasoning (NeurIPS 2026 · similarity 0.87)
• Better, Faster: Harnessing Self-Improvement in Large Reasoning Models (ICML 2026 · similarity 0.87)