Research map / Large language models / Reasoning and chain of thought
T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds T1, an RL-scaled reasoning model initialized from synthesized trial-and-error CoT SFT data, and proposes a truncation-based way to measure inference scaling in long single generations.
Problem
Existing LLM reasoning relies on imitation learning and RL yields modest gains; test-time scaling via repeated sampling needs external verifiers rather than longer single generations.
Approach
Scale RL for reasoning by encouraging exploration: SFT on synthesized CoT that embeds trial-and-error and self-verification, large oversampling (K=64) at high temperature, token-level entropy bonus, on-policy KL normalization with EMA reference, and -1 penalties for repetition/garbage; measure inference scaling by truncating a single long response and having the base model summarize the answer.
Key results
- T1 (Qwen2.5-32B) reaches 92.4 on MATH500, 50.6 on AIME2024, and 49.6 on Omni-MATH-500, versus 90.6, 50.0, and 46.6 for QwQ-32B-preview.
- T1 (Qwen2.5-14B) improves AIME2024 from 13.7 (Qwen2.5-14B-Instruct) to 30.5 and MATH500 from 78.9 to 87.4.
- Under truncated-thinking analysis, T1 (Qwen2.5-32B) summarized-answer accuracy on AIME rises from about 24% to 50% as thinking tokens increase.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| RL scaling (T1 RL training) on top of SFT initialization (ablation) | Qwen2.5-32B · AIME2024 | accuracy (%): 24.9 -> 50.6 | Helped |
| RL scaling (T1 RL training) on top of SFT initialization (ablation) | Qwen2.5-32B · MATH500 | accuracy (%): 83.4 -> 92.4 | Helped |
| RL scaling on top of SFT initialization (ablation) | Qwen2.5-14B · MATH500 | accuracy (%): 77.2 -> 87.4 | Helped |
| RL scaling on top of SFT initialization (ablation) | GLM-4-9B · MATH500 | accuracy (%): 60.2 -> 65.8 | Helped |
| Sampling temperature 0.9 vs 1.2 in RL (top-p only) (ablation) | Qwen2.5-14B RL training · MATH500 | accuracy (%): 78.2 -> 86.4 at temp 1.2 | Helped |
| Adding min-p 0.05 sampling at temperature 1.2 (ablation) | Qwen2.5-14B RL training · AIME2024 | accuracy (%): 29.3 -> 11.5 | Hurt |
| Adding min-p 0.05 sampling at temperature 1.2 (ablation) | Qwen2.5-14B RL training · MATH500 | accuracy (%): 86.4 -> 78.8 | Hurt |
| Higher sampling temperature (1.2 vs 0.9 or 1.3) in RL with top-p 0.95 (ablation) | Qwen2.5-14B RL training · Omni-MATH-500 | accuracy (%): 32.0 (t=0.9), 37.8 (t=1.1), 38.6 (t=1.2), 36.4 (t=1.3) | Mixed |
| Sampling 64 responses per prompt vs 4 in RL training (ablation) | Qwen2.5-14B · GPQA | accuracy gain: over 6% with 64 samples vs around 3% with 4 samples | Helped |
| Overlong/repetition/garbage-text penalty (reward -1) during RL (ablation) | Qwen2.5-14B RL training · MATH500 accuracy at step160 | accuracy (%): 81.2 with penalty vs 76.4 at step120 without | Helped |
| Truncated-thinking inference scaling with RL training steps (100% vs 30% RL steps) (ablation) | Qwen2.5-14B / Qwen2.5-32B · AIME2024 | accuracy (%): from 30% to 50% with 100% RL steps | Helped |
| Training with temperature <= 1.0 in RL (ablation) | Qwen2.5-14B RL training · training stability | training collapse | Hurt |
Surprising findings
- Training with temperature at or below 1.0 often collapsed after a few steps, while temperature 1.2 worked best.
- Min-p sampling made RL training more prone to collapse and worsened repetition, so only top-p with high temperature was used.
Limitations
- Training data is predominantly mathematical, with almost no science data.
- Inference scaling analysis uses the base model as a summarizer, a proxy rather than a true end-to-end budget control.
- Reward is binary correctness only; the paper does not test learned reward models.
Experimental setup
Datasets: MATH-train NuminaMath
Benchmarks: MATH500 AIME2024 Omni-MATH-500 GPQA
Metrics: Pass@1 accuracy
Baselines: GPT-4o Claude-3.5-sonnet Llama-3.3-70B-Instruct Qwen2.5-Math-7B-Instruct o1-preview QwQ-32B-preview GLM-4-9B-chat Qwen2.5-14B-Instruct Qwen2.5-32B-Instruct
Builds on: RLOO RLHF KL-regularized objective OpenAI o1 Qwen2.5-Math
Closest papers by meaning
- Incentivizing LLMs to Self-Verify Their Answers (NeurIPS 2025 · similarity 0.92)
- d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning (NeurIPS 2025 · similarity 0.92)
- Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement (ICML 2026 · similarity 0.92)
- RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning (ICLR 2026 · similarity 0.92)
- Beyond English-Centric Training: How Reinforcement Learning Improves Cross-Lingual Reasoning in LLMs (ICLR 2026 · similarity 0.91)
- Demystifying Long Chain-of-Thought Reasoning (ICML 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.