Research map / Large language models / Reasoning and chain of thought

Re-FORC: Adaptive Reward Prediction for Efficient Chain-of-Thought Reasoning

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Re-FORC trains lightweight Beta-distribution adapters on frozen Qwen3 reasoning models to forecast expected reward as a function of additional thinking tokens, and uses a Gittins-index / Pandora's box greedy policy for early stopping, model selection, and test-time scaling.

Problem

Determining how much inference-time reasoning compute to spend on a given query, and when to stop or switch models, requires knowing the marginal reward gain from additional thinking tokens, which current models do not predict.

Approach

Learn a forecaster on frozen reasoning-model hidden states that outputs Beta distributions over expected reward for a grid of future thinking-token budgets, then plug these forecasts into a Gittins-index / Pandora's box greedy policy with a per-token cost λ to decide when to stop, which model to use, and whether to sample again.

Key results

• Early stopping reduces compute by 26% on average vs. S1 for the Qwen3 4B model while preserving accuracy (abstract states up to 24-26% depending on version).
• Model selection with Re-FORC Pandora reaches 1.7 pp higher peak accuracy than All-8B while needing up to 12% less compute to match its accuracy.
• At peak compute, Re-FORC improves over the strongest DeepConf variant by 13.9 pp on 1.7B, 8.4 pp on 4B, and 7.5 pp on 8B.

Surprising findings

• The forecaster is occasionally overconfident, predicting higher rewards than the base model achieves, which leads to wasteful continued sampling.

Limitations

• Collecting forecaster training data is computationally expensive, requiring multiple trajectories per query at many length intervals.
• Forecaster occasionally overconfident, causing wasteful sampling instead of early termination.
• Evaluated only on math reasoning benchmarks with Qwen3 models up to 8B.

Experimental setup

Datasets: DeepScaleR-Preview AMC 2024 MATH500 AIME 2024 AIME 2025

Benchmarks: AMC 2024 MATH500 AIME 2024 AIME 2025

Metrics: Pearson correlation mean squared error mean absolute error accuracy vs. tokens accuracy vs. compute (TFLOPs)

Baselines: S1 DeepConf majority voting Avg-of-k Pass@k All 1.7B All 4B All 8B

Builds on: Pandora's box problem Gittins index policy S1 DeepConf

Closest papers by meaning

• DeepCompress: A Dual Reward Strategy for Dynamically Exploring and Compressing Reasoning Chains (ICLR 2026 · similarity 0.89)
• Reward Reasoning Models (NeurIPS 2025 · similarity 0.88)
• Fixing the Broken Compass: Diagnosing and Improving Inference-Time Reward Modeling (ICLR 2026 · similarity 0.88)
• RM-R1: Reward Modeling as Reasoning (ICLR 2026 · similarity 0.87)
• What If We Allocate Test-Time Compute Adaptively? (ICML 2026 · similarity 0.87)
• PEAR: Phase Entropy Aware Reward for Efficient Reasoning (ICLR 2026 · similarity 0.87)