Research map / Large language models / Agents, code and math
KALM: Knowledgeable Agents by Offline Reinforcement Learning from Large Language Model Rollouts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Grounds an LLM (Llama-2-7b-chat) in CLEVR-Robot via supervised instruction-following fine-tuning on environment rollouts, then uses it to generate imaginary rollouts for novel goals that augment offline RL (BC/CQL) training.
Problem
Offline RL agents are limited to skills present in their static dataset and generalize poorly to novel natural-language goals; there is also a semantic gap between LLM text tokens and numerical environment states/actions.
Approach
Replace the LLM's input/output layers with MLP embedding/output heads for states and actions, fine-tune it on four instruction-following tasks (dynamics prediction, rollout explanation, goal-to-rollout generation, consequence prediction), then prompt it to generate imaginary rollouts for novel goals and mix them with offline data for offline RL.
Key results
- KALM achieves 46% success on tasks with unseen goals versus 26% for baseline offline RL methods on CLEVR-Robot.
- Action generation accuracy stays around 30% while state generation accuracy improves over training, suggesting the LLM is better as an environment model than as a policy.
- Instruction-following fine-tuning (w/ IFF) outperforms no-IFF ablation, especially on unseen hard tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add LLM-generated imaginary rollouts to offline RL training | Offline RL agent · CLEVR-Robot unseen goals | success rate: 26% -> 46% | Helped |
| Add instruction-following prompts during LLM fine-tuning (ablation) | KALM · CLEVR-Robot unseen (hard) | success rate | Helped |
Surprising findings
- The LLM generates better states than actions, suggesting it is more suitable as an environment simulator than as a policy.
- Action generation accuracy stays roughly constant near 30% during training while state accuracy improves.
Limitations
- Evaluated only on the CLEVR-Robot environment with vector states, no visual observations.
- LLM must learn both dynamics and behavior policy for generated actions, adding burden.
- Performance degrades on hard tasks requiring reasoning about relative positions of multiple objects.
Experimental setup
Datasets: CLEVR-Robot offline dataset (100,000 rollout-goal pairs) LLM grounding set (400,000 trajectories)
Benchmarks: CLEVR-Robot
Metrics: success rate generation accuracy rollout explanation accuracy
Baselines: BC CQL
Builds on: Conservative Q-Learning Behavior cloning BERT MOBILE
Closest papers by meaning
- Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks (ICLR 2024 · similarity 0.87)
- Offline Training of Language Model Agents with Functions as Learnable Weights (ICML 2024 · similarity 0.87)
- Planning without Search: Refining Frontier LLMs with Offline Goal-Conditioned RL (NeurIPS 2025 · similarity 0.87)
- Agentic Reinforced Policy Optimization (ICLR 2026 · similarity 0.86)
- The Synergy of LLMs & RL Unlocks Offline Learning of Generalizable Language-Conditioned Policies with Low-fidelity Data (ICML 2025 · similarity 0.86)
- ML-Agent: Reinforcing LLM Agents for Autonomous Machine Learning Engineering (ICML 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.