AI Paper Atlas

Research map / Reinforcement learning / RL from feedback

TrajEvolve: Trajectory Evolution for Reinforcement Learning with Hindsight Credit Assignment

Yi Wen, Hao Chen, Wanyu Wang, Maolin Wang, Pengyue Jia, Derong Xu, Hui-Ze Tan, Yingyi Zhang, Wenlin Zhang, weihongluo, Xiku Du, Xiangyu Zhao

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: RL for LLM reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TrajEvolve, an RL method for LLM reasoning on long-horizon tasks that addresses sparse rewards and scarce successful trajectories via trajectory evolution and hindsight credit assignment.

Problem

RL for LLM reasoning on long-horizon tasks suffers from slow convergence because successful trajectories are rare, and sparse final rewards assign the same update to all steps.

Approach

Evolves trajectories to generate more successful samples and applies hindsight credit assignment so that individual steps receive differentiated credit rather than uniform updates from the final reward.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add trajectory evolution and hindsight step credit to GRPO-style RLQwen2.5-1.5B-Instruct · ALFWorldsuccess rate (%): GiGPO 86.7 -> TrajEvolve 92.2Improved
add trajectory evolution and hindsight step creditQwen2.5-7B-Instruct · WebShopsuccess rate (%): 68.8 -> 82.0 (vs GiGPO)Improved
remove the trajectory evolution module (ablation)TrajEvolve · ALFWorld and WebShopsuccess rateImproved
add step-level credit (binary necessity classification) (ablation)TrajEvolve · ALFWorld and WebShopsuccess rateImproved
add the evolution module to GRPO (ablation)GRPO · ALFWorld and WebShopsuccess rateImproved
add the evolution module to GiGPO (ablation)GiGPO · ALFWorld and WebShopsuccess rateImproved
use redundancy-aware reward and step-level advantage with evolutionQwen2.5-1.5B-Instruct policy · ALFWorldaverage steps: 15.36 -> 9.63 (GRPO vs Ours)Improved
use TrajEvolve instead of GiGPO on a Llama baseLlama-3.2-3B-Instruct · ALFWorldsuccess rate (%): 62.50 -> 92.19Improved
apply TrajEvolve to ScienceWorld with information-gathering steps retainedQwen2.5-7B-Instruct · ScienceWorld OODsuccess rate (%): 87.50 -> 89.06 (vs GiGPO)Improved
use TrajEvolve instead of GiGPO on ALFWorld Look subsetQwen2.5-1.5B-Instruct · ALFWorld Looksuccess rate (%): 95.0 -> 87.5Hurt
add hindsight verification and trajectory evolution overheadTrajEvolve training cost · per-step training timerelative time: approximately 1.5x of GRPOHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy