Research map / Large language models / Reasoning and chain of thought

Test-Time Scaling with Reflective Generative Model

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Reflective Generative Model (MetaStone-S1), sharing one backbone between policy and a lightweight self-supervised process reward head (SPRM) trained only with final-answer correctness, for best-of-k test-time scaling.

Problem

External test-time scaling relies on separate, costly process reward models that need step-level annotations and are off-policy; the paper aims to make reasoning trajectory selection cheap and label-free.

Approach

A Reflective Generative Form shares the backbone between the policy LLM and a Self-supervised Process Reward Model head (two linear layers plus dropout) that scores each step-token's hidden state; the head is trained with an SPR loss that uses only final-answer correctness and self-predicted pseudo-label filtering weights, enabling on-policy PRM learning with GRPO.

Key results

• MetaStone-S1-32B-high reaches 85.2 on AIME24 and 73.6 on AIME25, above QwQ-32B (79.5, 69.5) and close to o3-mini-medium (79.6, 74.8).
• SPRM with ~26M extra parameters on 7B (70.2 AIME24) outperforms a 72B Qwen2.5-Math-PRM (68.8) on AIME24.
• MetaStone-S1-1.5B gains 18.6 points on AIME24 over its baseline (57.9 vs 39.3 at high effort).

Surprising findings

• Outcome-only self-supervised SPRM with 5-53M extra parameters matches or beats 72B external reward models on AIME24/AIME25/LiveCodeBench.
• Treating final-answer correctness as step-level labels adds noise; the self-filtered SPR loss gives a larger score gap than BCE.

Limitations

• MCTS with SPRM underperforms Best-of-N under the reported budget due to search overhead.
• Training data limited to math-related sources; generalization to other domains is only shown on LiveCodeBench and C-Eval.
• Reported o3-mini numbers are marked with an asterisk and the comparison mixes effort levels.

Experimental setup

Datasets: NuminaMath OpenR1-Math-220k DeepScaleR LIMR OREAL-RL

Benchmarks: AIME2024 AIME2025 LiveCodeBench C-Eval

Metrics: Pass@1 averaged over 64 repeats

Baselines: Qwen2.5-Math-RM Qwen2.5-Math-PRM DeepSeek-R1-671B OpenAI o3-mini QwQ-32B R1-Distill-Qwen-32B DeepScaleR-1.5B-Preview GLM-Z1-32B-0414

Builds on: GRPO DeepSeek-R1-Distill-Qwen QwQ-32B

Closest papers by meaning

• GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning (AAAI 2026 · similarity 0.86)
• Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models (NeurIPS 2025 · similarity 0.85)
• SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning (ICML 2026 · similarity 0.85)
• Rethinking Reward Models for Multi-Domain Test-Time Scaling (NeurIPS 2026 · similarity 0.85)
• From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling (AAAI 2026 · similarity 0.85)
• Beyond What Seems Necessary: Hidden Gains from Scaling Training-Time Reasoning Length under Outcome Supervision (NeurIPS 2026 · similarity 0.85)