Research map / Reinforcement learning / RL from feedback

Optimizing Test-Time Compute via Meta Reinforcement Finetuning

ICML 2025 · Reinforcement learning › RL from feedback · approach: Scaling RL with verifiable rewards

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Meta Reinforcement Fine-Tuning (MRT), which frames test-time compute use as meta-RL, minimizes cumulative regret via a dense progress reward bonus added to outcome-reward RL or STaR, and improves math reasoning accuracy and token efficiency.

Problem

Outcome-reward RL with 0/1 rewards trains LLMs to produce long reasoning traces, but it is unclear whether these models use test-time compute efficiently and keep making progress as the token budget grows.

Approach

Segment the output stream into episodes, treat test-time generation as an in-context RL adaptation procedure, and minimize cumulative regret by adding a dense reward bonus equal to the progress (change in the likelihood that a meta-prover policy reaches the correct answer) of each episode.

Key results

• MRT on DeepScaleR-1.5B-Preview reaches 55.9 average pass@1 vs 55.0 for GRPO and 54.5 base, with AIME 2024 47.2 vs 44.5 GRPO.
• MRT (RL) needs about 5x fewer tokens than the base R1-Distill-Qwen-1.5B to match its AIME 2024 performance and improves over GRPO by 1.2-1.6x in token efficiency.
• On backtracking search with Llama, MRT achieves 1.6-1.7x token efficiency over STaR and GRPO in linearized evaluation.

Measured effects reported in the paper

Surprising findings

• Current outcome-reward-trained R1-distilled models often do not improve accuracy with more episodes, and sometimes degrade, on problems requiring many episodes.
• Majority voting over fewer episodes can outperform long sequential chains of thought on harder problems.

Limitations

• Choice of meta-prover mu is heuristic (best-guess answer forced after thinking) and may not be optimal.
• Base models restricted to a narrow set of strategies, limiting gains.
• Dense reward computed at end of trace rather than via branched rollouts, increasing gradient variance; higher train-time compute required.

Experimental setup

Datasets: NuminaMath AIME 1989-2023 Omni-MATH MATH

Benchmarks: AIME 2024 AIME 2025 AMC 2023 MinervaMATH MATH500

Metrics: pass@1 maj@K accuracy vs tokens token efficiency normalized cumulative regret

Baselines: outcome-reward RL (GRPO) STaR length penalty RISE base model

Builds on: GRPO STaR RISE budget forcing

Closest papers by meaning

• Scaling Test-Time Compute Without Verification or RL is Suboptimal (ICML 2025 · similarity 0.91)
• TTRL: Test-Time Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
• On the Limits of Test-Time Compute: Sequential Reward Filtering for Better Inference (ICML 2026 · similarity 0.90)
• Compute Aligned Training: Optimizing for Test Time Inference (NeurIPS 2026 · similarity 0.90)
• e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMs (ICLR 2026 · similarity 0.90)
• Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.89)