Research map / Large language models / Reasoning and chain of thought

Tool Verification for Test-Time Reinforcement Learning

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes T3RL, a test-time RL method that weights majority-vote pseudo-labels by code-execution-based verification of rollouts, improving over TTRL on math benchmarks.

Problem

Test-time RL with majority-vote self-consensus rewards can reinforce frequent but wrong answers (false-popular mode collapse) on unlabeled test inputs.

Approach

An LLM verifier compiles each rollout into Python, a code interpreter executes it, and rollouts whose executed result matches the extracted answer receive an upweighted vote (omega) in a verification-aware weighted majority vote that sets the GRPO pseudo-label and binary reward.

Key results

• On AIME 2024, Qwen2.5-Math-1.5B improves from 15.8 (TTRL) to 20.8 with T3RL, a +31.6% relative gain; T3RL wins all 15 paired model-benchmark comparisons against TTRL.
• On MATH-500 difficulty level L5, T3RL reaches 49.0 vs 47.0 for TTRL (+4.3% relative).
• T3RL with 16 rollouts surpasses TTRL with 64 rollouts while using about 60% fewer training tokens and 61% less wall-clock time.

Surprising findings

• A 0.5B verifier degrades performance below TTRL (avg 14.3 vs 15.7), as small verifiers copy the trace's answer instead of recomputing.
• Hard filtering (omega to infinity) is less reliable than a finite weight, and the self-verifier (policy as verifier) also beats TTRL.

Limitations

• Verifier quality limits gains: a 0.5B verifier underperforms TTRL, and Coder-1.5B verifier accuracy is only 73.2%.
• Gains are small on easy tasks where rollouts already agree.
• Verification weight omega is a hand-tuned hyperparameter, and the non-math generalization is shown on a single small experiment.

Experimental setup

Datasets: AIME 2024 AMC MATH-500 Knights-and-Knaves puzzles

Benchmarks: AIME 2024 AMC MATH-500 Knights-and-Knaves

Metrics: pass@1 accuracy standard deviation across runs tokens wall-clock time TFLOPs

Baselines: TTRL TTRL-Agent Buffer-of-Thoughts BoN-RR T1 T1+PRM

Builds on: TTRL GRPO tool-integrated reasoning

Closest papers by meaning

• TTRL: Test-Time Reinforcement Learning (NeurIPS 2025 · similarity 0.92)
• Scaling Test-Time Compute Without Verification or RL is Suboptimal (ICML 2025 · similarity 0.88)
• Meta-TTRL: A Metacognitive Framework for Self-Improving Test-Time Reinforcement Learning for T2I Generation in Unified Multimodal Models (NeurIPS 2026 · similarity 0.88)
• From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generation (ICLR 2026 · similarity 0.87)
• Mode-conditioning unlocks superior test-time compute scaling (ICLR 2026 · similarity 0.87)
• Optimizing Test-Time Compute via Meta Reinforcement Finetuning (ICML 2025 · similarity 0.87)