Research map / Reinforcement learning / RL from feedback
Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TBRM, an off-policy value-based RL method that regresses LLM logits onto a trajectory-level Bellman residual, with a convergence proof, and matches or beats PPO and GRPO on math reasoning benchmarks.
Problem
Policy-based methods (PPO, GRPO) dominate RL for LLM reasoning and need on-policy rollouts, critics, clipping, and multiple samples per prompt, while value-based methods for LLMs are underexplored.
Approach
Lift Bellman Residual Minimization from the token level to the trajectory level: square a single residual over the whole rollout and regress the model's logits (used as Q-values) onto it, using a change-of-trajectory-measure lemma so that only state-wise distribution shift matters, enabling off-policy training with one rollout per prompt.
Key results
- TBRM with n=4 reaches 30.5% on AIME24 with Qwen2.5-Math-7B, exceeding GRPO n=4 (28.9%).
- TBRM n=1 uses 33.3% less GPU memory than PPO n=1 and gives a 1.3x speedup on Qwen2.5-Math-7B.
- TBRM n=1 gets 3.1x training speedup over GRPO n=4 while trailing it by only 1.27% average on 7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TBRM n=1 instead of PPO n=1 | Qwen2.5-Math-1.5B · AIME24 Avg@32 | accuracy: 13.2 vs 11.4 | Helped |
| Use TBRM n=1 instead of PPO n=1 | Qwen2.5-Math-1.5B · MATH500 Avg@1 | accuracy: 72.2 vs 72.2 | No clear effect |
| Increase TBRM rollouts per prompt from n=1 to n=4 (ablation) | Qwen2.5-Math-1.5B · AMC23 Avg@32 | accuracy: 48.6 -> 52.0 | Helped |
| Increase TBRM rollouts per prompt from n=1 to n=4 (ablation) | Qwen2.5-Math-1.5B · OlympiadBench Avg@1 | accuracy: 35.7 -> 36.1 | Mixed |
| Use TBRM n=4 instead of GRPO n=4 | Qwen2.5-Math-1.5B · AIME24 Avg@32 | accuracy: 14.3 vs 13.0 | Helped |
| Use TBRM n=4 instead of GRPO n=4 | Qwen2.5-Math-1.5B · AIME25 Avg@32 | accuracy: 6.9 vs 7.1 | Hurt |
| Increase TBRM rollouts per prompt from n=1 to n=4 (ablation) | Qwen2.5-Math-7B · AIME24 Avg@32 | accuracy: 24.1 -> 30.5 | Helped |
| Use TBRM n=4 instead of GRPO n=4 | Qwen2.5-Math-7B · OlympiadBench Avg@1 | accuracy: 44.1 vs 42.5 | Helped |
| Use TBRM n=1 instead of PPO n=1 | Qwen2.5-Math-7B · AIME24 Avg@32 | accuracy: 24.1 vs 25.4 | Hurt |
| Use TBRM n=1 instead of GRPO n=4 | Qwen2.5-Math-7B · AMC23 Avg@32 | accuracy: 63.4 vs 66.8 | No clear effect |
| Use TBRM n=1 instead of GRPO n=4 | Qwen2.5-Math-1.5B · AIME25 Avg@32 | accuracy: 5.6 vs 7.1 | Mixed |
| Train with TBRM n=1 instead of PPO n=1 | Qwen2.5-Math-7B · training step time | speedup: 1.3× | Helped |
Surprising findings
- Directly applying token-level BRM to LLMs leads to unstable training and reward collapse, per the authors' experiments.
Limitations
- Theory relies on realizability and bounded, smooth parametrization assumptions that may not hold for LLMs.
- Experiments limited to math reasoning with Qwen2.5-Math models; non-math results only briefly mentioned.
- Beta fixed at 0.002 across experiments with limited sensitivity analysis described.
Experimental setup
Datasets: DAPO prompt set (~17.4k math problems)
Benchmarks: AIME24 AIME25 AMC23 MATH500 Minerva Math OlympiadBench
Metrics: Avg@32 Avg@1 training time per step peak GPU memory
Baselines: PPO GRPO
Builds on: Bellman Residual Minimization KL-regularized RL PPO GRPO change-of-trajectory-measure lemma
Closest papers by meaning
- Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.92)
- ShiQ: Bringing back Bellman to LLMs (NeurIPS 2025 · similarity 0.91)
- Efficient Reinforcement Learning with Large Language Model Priors (ICLR 2025 · similarity 0.90)
- STRIDE: Learnable Stepwise Language Feedback for LLM Reasoning (NeurIPS 2026 · similarity 0.90)
- Planning without Search: Refining Frontier LLMs with Offline Goal-Conditioned RL (NeurIPS 2025 · similarity 0.90)
- SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoning (ICML 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.