Research map / Large language models / Reasoning and chain of thought

TTRL: Test-Time Reinforcement Learning

NeurIPS 2025 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TTRL, which runs GRPO reinforcement learning on unlabeled test questions using majority-voted pseudo-labels from repeated sampling as rule-based rewards, improving reasoning models on math and GPQA benchmarks.

Problem

Reinforcement learning for LLM reasoning normally needs labeled data, and test-time data has no ground-truth labels, so reward estimation at inference is the core challenge.

Approach

Estimate a pseudo-label for each test question by majority voting over many sampled rollouts, then convert agreement with that label into binary rule-based rewards and run online RL (GRPO) at test time; the reward signal is self-generated and needs no ground truth.

Key results

• On Qwen2.5-Math-7B, TTRL raises AIME 2024 pass@1 from 12.9 to 40.2 (+211.6%) with only unlabeled test data.
• Average gain across AIME 2024, AMC, MATH-500 and GPQA for Qwen2.5-Math-7B is 76.5% (31.1 to 54.9).
• TTRL avg@64 on Qwen2.5-Math-7B consistently exceeds the initial model's maj@64 across benchmarks, surpassing the self-training upper bound.

Measured effects reported in the paper

Surprising findings

• Reward accuracy stays above 75% even when label accuracy rarely exceeds 50%, attributed to scattered incorrect predictions (the 'Lucky Hit' effect).
• TTRL trained on test data surpasses the initial model's majority-vote accuracy despite being supervised only by that majority vote.

Limitations

• Performance depends on prior knowledge of the backbone; gains shrink on harder MATH-500 difficulty levels and TTRL can fail when priors are insufficient.
• Sensitive to RL hyperparameters such as temperature and number of episodes; poor settings cause persistently high entropy and failed training.
• Some comparisons with prior R1-Zero-like models use a different setup, making comparisons unfair as acknowledged by the authors.

Experimental setup

Datasets: AIME 2024 AMC MATH-500 GPQA-Diamond

Benchmarks: AIME 2024 AMC MATH-500 GPQA-Diamond

Metrics: pass@1 avg@16 maj@16 maj@64 accuracy

Baselines: Qwen2.5-Math-7B (backbone) Qwen2.5-Math-7B-Instruct Oat-Zero-7B SimpleRL-Zero-7B PRIME-Zero-7B RL (leakage)

Builds on: GRPO PPO PRIME test-time scaling self-consistency

Closest papers by meaning

• Tool Verification for Test-Time Reinforcement Learning (NeurIPS 2026 · similarity 0.92)
• T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling (ICML 2025 · similarity 0.91)
• ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models (NeurIPS 2025 · similarity 0.91)
• Optimizing Test-Time Compute via Meta Reinforcement Finetuning (ICML 2025 · similarity 0.91)
• On Predictability of Reinforcement Learning Dynamics for Large Language Models (ICLR 2026 · similarity 0.91)
• Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.90)