Research map / Large language models / Instruction and fine-tuning
Learning to Think: Information-Theoretic Reinforcement Fine-Tuning for LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Learning to Think (L2T), an RL fine-tuning framework that splits reasoning into episodes and rewards each episode with an information-theoretic dense process reward (fitting information gain minus a PAC-Bayes/Fisher-based parameter compression penalty), built on GRPO.
Problem
LLM reasoning RL with sparse outcome rewards encourages unnecessarily long chains of thought, wasting tokens, while existing process-reward methods need costly task-specific annotations or evaluators.
Approach
A universal dense process reward computed from internal model signals: the change in predicted correctness probability (fitting gain) minus a parameter compression penalty approximated via low-rank SVD proxies, Gaussian/PAC-Bayes mutual-information approximation and the Fisher information matrix; no extra annotations are needed.
Key results
- On DeepScaleR-1.5B-Preview, L2T averages 57.8 across five math benchmarks vs 55.0 for GRPO and 55.9 for MRT (base 54.5).
- On DeepSeek-R1-Distill-Qwen-1.5B, L2T averages 49.2 vs 46.0 for GRPO and 47.1 for MRT (base 44.9).
- Authors report L2T uses about 18% of base model tokens and roughly 50% of outcome-reward fine-tuning tokens.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune with L2T instead of outcome-reward GRPO | DeepScaleR-1.5B-Preview · Math reasoning benchmarks (average) | Pass@1: 55.0 (+0.5) -> 57.8 (+3.3) | Helped |
| Fine-tune with L2T instead of outcome-reward GRPO | DeepSeek-R1-Distill-Qwen-1.5B · Math reasoning benchmarks (average) | Pass@1: 46.0 (+1.1) -> 49.2 (+4.3) | Helped |
| Use L2T instead of outcome-reward fine-tuning | LLMs evaluated on AIME and other benchmarks · AIME and other reasoning benchmarks | token efficiency: roughly doubles token efficiency | Helped |
| Use L2T instead of process-reward methods | LLMs evaluated on reasoning benchmarks · Reasoning benchmarks | token budget: reducing the token budget by 20% | Helped |
| Replace fitting information gain with a task-specific reward model (ablation) | L2T · Reasoning benchmarks | reasoning performance | Hurt |
| Remove the parameter-compression penalty (ablation) | L2T · Reasoning benchmarks | reasoning performance | Hurt |
| Replace low-rank approximation with random sampling of 30% layers (ablation) | L2T · Reasoning benchmarks | approximation error | Hurt |
| Set α=0.8 and β=0.6 rather than other swept settings (ablation) | L2T · Reasoning benchmarks | average outcome | Helped |
Surprising findings
- Existing outcome-reward RL often lets models use more than twice the tokens needed for correct answers, and extra episodes can degrade accuracy on simple Tier 1 questions.
Limitations
- The Gaussian and small-step approximations behind the compression penalty are heuristic and rely on assumptions (e.g., Sigma_s approx Sigma_0, CLT argument).
- Fine-tuning uses small training sets (919 AIME questions, 4,000 NuminaMath samples) and only 1.5B and 7B base models were evaluated in the main text.
Experimental setup
Datasets: AIME (1989-2023) NuminaMath Omni-MATH MATH500 MinervaMATH AMC 2023 AIME 2024 AIME 2025
Benchmarks: AIME 2024 AIME 2025 AMC 2023 MATH500 MinervaMATH Omni-MATH
Metrics: pass@1 token budget
Baselines: GRPO length penalty ReST-MCTS MRT
Builds on: GRPO PAC-Bayes bounds Fisher information matrix ReST-MCTS MRT
Closest papers by meaning
- Thinker: Learning to Think Fast and Slow (NeurIPS 2025 · similarity 0.90)
- Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.90)
- RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning (ICLR 2026 · similarity 0.90)
- Thinkless: LLM Learns When to Think (NeurIPS 2025 · similarity 0.89)
- Boosting LLM Reasoning via Human-Inspired Reward Shaping (NeurIPS 2026 · similarity 0.89)
- Thinking on the Fly: Test-Time Reasoning Enhancement via Latent Thought Policy Optimization (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.