Research map / Reinforcement learning / Visual and representation RL
The Synergy of LLMs & RL Unlocks Offline Learning of Generalizable Language-Conditioned Policies with Low-fidelity Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TEDUO, a three-step pipeline that uses LLM hindsight labeling and state abstraction to make offline RL labels from unlabeled transitions, then distills the learned policies into a Llama-3-8B via SFT to get a language-conditioned policy that generalizes to novel goals and environments.
Problem
Learning language-conditioned policies offline from unlabeled, non-expert state-action transitions and an unpaired list of goals, with generalization to unseen language commands and states, where labeled data and online interaction are scarce.
Approach
Use LLMs in two roles: as cheap labelers (prompted goal-completion detection distilled into lightweight proxy reward networks, plus LLM-guided state abstraction) to turn unlabeled data into per-goal offline RL problems solved with tabular Q-learning, and then as generalizable policies by fine-tuning a pretrained LLM on the optimal action sequences produced by those policies.
Key results
- TEDUO-Llama-3-8B raises success rate from 17% (vanilla Llama-3-8B) to 65% on training goals and 45% on test goals and test environments in BabyAI Synth.
- BabyAI-IL-bot drops about 40% on new goals while TEDUO drops about 8% (success 55% vs 15% for unseen goals on training environments).
- On WebShop, TEDUO-Llama-3-8B reaches score 44.4 on testing instructions versus 8.4 for ReAct-Llama-3-8B and 13.8 for ReAct-Llama-3-70B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Llama-3-8B with TEDUO instead of using the vanilla model | Llama-3-8B · BabyAI Synth, train environments + train goals | success rate: 17 to 65 | Helped |
| Use TEDUO-Llama-3-8B instead of BabyAI-IL-bot for test environments and test goals | BabyAI instruction following · BabyAI Synth, test environments + test goals | success rate: 16 to 45 | Helped |
| Add offline Q-learning (step 2) instead of using goal-conditioned behavioral cloning (ablation) | TEDUO steps 1 and 2 · BabyAI Synth, train goals | success rate: 7 to 16 | Helped |
| Add LLM supervised fine-tuning (step 3) to the GCRL policy (ablation) | TEDUO · BabyAI Synth, train goals | success rate: 16 to 65 | Helped |
| Replace tabular Q-learning with DQN in step 2 (ablation) | TEDUO steps 1 and 2 · BabyAI Synth, train goals | success rate: 16 (±0.8) to 15 (±1.2) | Hurt |
| Train on both environment types A and B instead of only one type | TEDUO · BabyAI type C test tasks | success rate: 0 to 60 | Helped |
| Fine-tune Llama-3-8B with TEDUO instead of ReAct prompting | Llama-3-8B · Webshop, training goals | score: 8.4 to 52.1 | Helped |
| Add offline Q-learning steps 1 and 2 instead of using the LLM-random data collection policy (ablation) | TEDUO · Webshop, training goals | score: 5.6 to 56.2 | Helped |
| Add LLM supervised fine-tuning (step 3) to the GCRL policy (ablation) | TEDUO · Webshop, training goals | score: 56.2 to 52.1 | Hurt |
| Increase the number of training goals from 266 to 534 | TEDUO-Llama-3-8B · BabyAI Synth, unseen test goals | success rate: 33 to 45 | Helped |
| Apply LLM-based state abstraction during step 1 (ablation) | TEDUO Q-learning policies · BabyAI Synth | data efficiency | Helped |
Surprising findings
- Non-fine-tuned Llama-3-70B and DeepSeek-R1 distilled models perform poorly on BabyAI, with success rates of 14-32%, despite their scale.
- A TEDUO agent trained only on type A or type B environments achieves 0% on type C tasks, but training on both achieves 60% on type C, suggesting skill composition.
Limitations
- Requires textual state representation and discrete action space, limiting continuous control and non-textual observations.
- Assumes practitioner knowledge to propose the goal list and simple goals representable as state subsets.
- Tabular Q-learning in step 2 does not generalize to new transitions; performance limited by data collection coverage (initial states not visited).
Experimental setup
Datasets: BabyAI Synth environments (generated transitions) WebShop
Benchmarks: BabyAI Synth (online generalization) WebShop
Metrics: success rate episode length invalid actions score
Baselines: Llama-3-8B vanilla Llama-3-70B vanilla Llama-3-8B in-context+CoT Llama-3-70B in-context+CoT DeepSeek-R1 distilled to Llama-8B BabyAI-IL-bot Step 1 + GCBC Steps 1+2 GCRL ReAct-Llama-3-8B ReAct-Llama-3-70B
Builds on: tabular Q-learning BabyAI WebShop ReAct
Closest papers by meaning
- Planning without Search: Refining Frontier LLMs with Offline Goal-Conditioned RL (NeurIPS 2025 · similarity 0.90)
- RLZero: Direct Policy Inference from Language Without In-Domain Supervision (NeurIPS 2025 · similarity 0.89)
- Reinforcement World Model Learning for LLM-based Agents (NeurIPS 2026 · similarity 0.89)
- Divide and Conquer: Grounding LLMs as Efficient Decision-Making Agents via Offline Hierarchical Reinforcement Learning (ICML 2025 · similarity 0.89)
- Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learning (ICLR 2024 · similarity 0.89)
- Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.