Research map / Large language models / Agents, code and math
Policy Improvement using Language Feedback Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Language Feedback Models (LFMs), small Flan-T5 models distilled from GPT-4 feedback on verbalized trajectories, which identify desirable actions for imitation-based policy improvement on ALFWorld, ScienceWorld, and Touchdown.
Problem
Instruction-following agents in grounded environments are sample-inefficient and poorly generalizable, and using LLMs online as experts for action prediction is expensive and inaccurate.
Approach
Use an LLM offline to label which steps of verbalized behaviour-cloned rollouts are productive, distill these labels into a small feedback model, then use that model online to select desirable actions for imitation, avoiding LLM calls during policy improvement.
Key results
- LFM imitation reaches 64.1% on ALFWorld, 47.1% on ScienceWorld, and 59.7% on Touchdown, above behaviour cloning (62.6, 45.8, 57.5) under the same 100k LLM token budget.
- Adapting with LFM for one round on unseen test environments gives 74.6% (ALFWorld), 49.3% (ScienceWorld), and 62.8% (Touchdown).
- The learned LFM reaches F1 of 93.2 (ALFWorld), 83.7 (ScienceWorld), and 43.9 (Touchdown) against GPT-4 labels.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use LFM-identified desirable behavior instead of behavior cloning alone | Behavior-cloned policy · ALFWorld | task-completion rate: 62.6 -> 64.1 | Helped |
| use LFM-identified desirable behavior instead of behavior cloning alone | Behavior-cloned policy · ScienceWorld | task-completion rate: 45.8 -> 47.1 | Helped |
| use LFM-identified desirable behavior instead of behavior cloning alone | Behavior-cloned policy · Touchdown | task-completion rate: 57.5 -> 59.7 | Helped |
| use LFM feedback instead of LLM action prediction, with the same output-token budget | Policy improvement with GPT-4 · ALFWorld | task-completion rate: 56.0 -> 64.1 | Helped |
| use LFM feedback instead of LLM action prediction, with the same output-token budget | Policy improvement with GPT-4 · ScienceWorld | task-completion rate: 39.0 -> 47.1 | Helped |
| use LFM feedback instead of LLM action prediction, with the same output-token budget | Policy improvement with GPT-4 · Touchdown | task-completion rate: 58.0 -> 59.7 | Helped |
| adapt the policy using LFM-identified behavior in test environments for one round | LFM policy · ALFWorld | task-completion rate: 64.1 -> 74.6 | Helped |
| adapt the policy using LFM-identified behavior in test environments for one round | LFM policy · ScienceWorld | task-completion rate: 47.1 -> 49.3 | Helped |
| adapt the policy using LFM-identified behavior in test environments for one round | LFM policy · Touchdown | task-completion rate: 59.7 -> 62.8 | Helped |
| provide detailed explanations and feedback instead of succinct feedback (ablation) | LFM · ALFWorld | F1 score: 93.2 -> 92.0 | No clear effect |
| provide detailed explanations and feedback instead of succinct feedback (ablation) | LFM · ScienceWorld | F1 score: 83.7 -> 82.5 | No clear effect |
| provide detailed explanations and feedback instead of succinct feedback (ablation) | LFM · Touchdown | F1 score: 43.9 -> 42.5 | No clear effect |
Surprising findings
- Llama 2 70B feedback produced many spurious desirable labels and lowered task completion relative to GPT-4 feedback (e.g., 64.1% to 56.0% on ALFWorld).
- DAgger with LLM action prediction underperformed simple LLM action prediction on all benchmarks, dropping to 22.5% on ScienceWorld.
Limitations
- Relies on the verbalization module faithfully describing observations; verbalization is hard for richer scenes.
- LLM feedback can hallucinate, and weaker LLMs such as Llama 2 give poor feedback.
- Small 250M Flan-T5 variant shows roughly 10% task-completion drop versus 770M, so policies still need large GPUs to train.
Experimental setup
Datasets: ALFWorld ScienceWorld Touchdown
Benchmarks: ALFWorld ScienceWorld Touchdown
Metrics: task-completion rate F1 score of productive/not-productive predictions
Baselines: Behaviour cloning LLM action prediction imitation learning DAgger GPT-4 zeroshot Prev SOTA on each benchmark
Builds on: DAgger ALFWorld Touchdown ScienceWorld
Closest papers by meaning
- Interpreting Learned Feedback Patterns in Large Language Models (NeurIPS 2024 · similarity 0.89)
- Learning with Language-Guided State Abstractions (ICLR 2024 · similarity 0.89)
- Inner Speech as Behavior Guides: Steerable Imitation of Diverse Behaviors for Human-AI coordination (NeurIPS 2025 · similarity 0.88)
- Hypothesis, Verification, and Induction: Grounding Large Language Models with Self-Driven Skill Learning (AAAI 2024 · similarity 0.88)
- Aligning Language Models with Demonstrated Feedback (ICLR 2025 · similarity 0.88)
- Do LLMs ``know'' internally when they follow instructions? (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.