Research map / Multi-agent systems and games / LLM agents

Spinning Straw into Gold: Relabeling LLM Agent Trajectories in Hindsight for Successful Demonstrations

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Self-evolving language agent learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Hindsight Supervised Learning (HSL), which uses a zero-shot Llama-3.3-70B relabeler to mine all goals an LLM agent actually achieved in its rollouts, then fine-tunes a Llama-3.2-1B agent on the relabeled demos with irrelevant-action masking and reweighting, improving SFT and DPO on ALFWorld, PlanCraft and WebShop.

Problem

LLM agents in partially observable, long-horizon settings lack sufficient supervision; expert demonstrations are costly and SFT does not exploit the agent's own rollouts, while RL struggles with sparse rewards.

Approach

Treat unintended but successful achievements in agent rollouts as valid goals: an auxiliary LLM relabels each trajectory with all natural-language goals achieved, pairs them with relevance-labeled trajectory segments, and continually refreshes this on-policy relabeled buffer for SFT, with action masking and demo reweighting to handle suboptimal relabeled data.

Key results

• SFT+HSL reaches 93.57% (seen) and 96.27% (unseen) on ALFWorld across three seeds, versus 79.52% and 75.87% for SFT alone.
• DPO+HSL reaches 70.52 task score on WebShop and 75.42% on PlanCraft, the best results in the main table.
• DPO+HSL with 800 ground-truth demos reaches 92.5% on ALFWorld unseen, above DPO with over 3,200 demos (82.8%).

Surprising findings

• SelfImit, which fine-tunes on the agent's own successful trajectories, fails to improve over SFT.
• ReAct and BehaviorClone using the same 70B model perform substantially worse than SFT+HSL, indicating relabeling is easier than acting.

Limitations

• Relies on zero-shot external 70B relabeler; relabeling quality may not hold for more complex tasks, and joint training is left for future work.
• Binary action relevance labels ignore actions that are relevant but suboptimal.
• Still requires ground-truth demonstrations for stable training and predefined goal spaces; only Llama-3 base agents tested.

Experimental setup

Datasets: ALFWorld PlanCraft WebShop

Benchmarks: ALFWorld PlanCraft WebShop

Metrics: success rate task score

Baselines: ReAct BehaviorClone Bagel SelfImit SFT DPO

Builds on: hindsight experience replay ETO GCSL DPO

Closest papers by meaning

• AgentRefine: Enhancing Agent Generalization through Refinement Tuning (ICLR 2025 · similarity 0.88)
• Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Tasks (NeurIPS 2025 · similarity 0.88)
• A Subgoal-driven RL Framework for Improving Long-Horizon Web Agents (NeurIPS 2026 · similarity 0.88)
• Latent Action Reparameterization for Efficient Agent Inference (NeurIPS 2026 · similarity 0.88)
• Learning Agentic Policy from Action Guidance (NeurIPS 2026 · similarity 0.87)
• How to Train Your LLM Web Agent: A Statistical Diagnosis (NeurIPS 2025 · similarity 0.87)