Research map / Reinforcement learning / RL from feedback
TrajEvolve: Trajectory Evolution for Reinforcement Learning with Hindsight Credit Assignment
Yi Wen, Hao Chen, Wanyu Wang, Maolin Wang, Pengyue Jia, Derong Xu, Hui-Ze Tan, Yingyi Zhang, Wenlin Zhang, weihongluo, Xiku Du, Xiangyu Zhao
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TrajEvolve, an RL method for LLM reasoning on long-horizon tasks that addresses sparse rewards and scarce successful trajectories via trajectory evolution and hindsight credit assignment.
Problem
RL for LLM reasoning on long-horizon tasks suffers from slow convergence because successful trajectories are rare, and sparse final rewards assign the same update to all steps.
Approach
Evolves trajectories to generate more successful samples and applies hindsight credit assignment so that individual steps receive differentiated credit rather than uniform updates from the final reward.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add trajectory evolution and hindsight step credit to GRPO-style RL | Qwen2.5-1.5B-Instruct · ALFWorld | success rate (%): GiGPO 86.7 -> TrajEvolve 92.2 | Improved |
| add trajectory evolution and hindsight step credit | Qwen2.5-7B-Instruct · WebShop | success rate (%): 68.8 -> 82.0 (vs GiGPO) | Improved |
| remove the trajectory evolution module (ablation) | TrajEvolve · ALFWorld and WebShop | success rate | Improved |
| add step-level credit (binary necessity classification) (ablation) | TrajEvolve · ALFWorld and WebShop | success rate | Improved |
| add the evolution module to GRPO (ablation) | GRPO · ALFWorld and WebShop | success rate | Improved |
| add the evolution module to GiGPO (ablation) | GiGPO · ALFWorld and WebShop | success rate | Improved |
| use redundancy-aware reward and step-level advantage with evolution | Qwen2.5-1.5B-Instruct policy · ALFWorld | average steps: 15.36 -> 9.63 (GRPO vs Ours) | Improved |
| use TrajEvolve instead of GiGPO on a Llama base | Llama-3.2-3B-Instruct · ALFWorld | success rate (%): 62.50 -> 92.19 | Improved |
| apply TrajEvolve to ScienceWorld with information-gathering steps retained | Qwen2.5-7B-Instruct · ScienceWorld OOD | success rate (%): 87.50 -> 89.06 (vs GiGPO) | Improved |
| use TrajEvolve instead of GiGPO on ALFWorld Look subset | Qwen2.5-1.5B-Instruct · ALFWorld Look | success rate (%): 95.0 -> 87.5 | Hurt |
| add hindsight verification and trajectory evolution overhead | TrajEvolve training cost · per-step training time | relative time: approximately 1.5x of GRPO | Hurt |
Closest papers by meaning
- Prioritize the Process, Not Just the Outcome: Rewarding Latent Thought Trajectories Improves Reasoning in Looped Language Models (ICML 2026 · similarity 0.89)
- On Predictability of Reinforcement Learning Dynamics for Large Language Models (ICLR 2026 · similarity 0.89)
- TTRL: Test-Time Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMs (NeurIPS 2025 · similarity 0.88)
- ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models (NeurIPS 2025 · similarity 0.88)
- Not Only Where, But When: Temporal Scheduling for RLVR (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
