Research map / Reinforcement learning / RL from feedback
Efficient Reinforcement Learning with Large Language Model Priors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Treats a fixed LLM as an action prior inside RL via variational inference and direct posterior sampling, yielding DQN-Prior, CQL-Prior and GFlan-Prior variants that cut sample requirements on ALFWorld and Overcooked.
Problem
Reinforcement learning for text-based sequential decision-making needs extensive exploration and generalizes poorly, while LLM agents rely on costly prompting or fine-tuning.
Approach
Model the LLM as a prior over actions in a Bayesian/control-as-inference framework; sample k candidate actions from the frozen LLM and restrict exploration, Bellman backups, and CQL penalties to this prior action subspace, or use a KL term to the LLM prior for policy-based PPO.
Key results
- CQL-Prior reaches 1.01 on Overcooked(Salad-1000) versus 0.32 for CQL, and 0.80 on ALFWorld(Pick) versus 0.03 for CQL.
- The paper reports CQL-Prior needs about 1000 samples versus more than 12000 for CQL, a reduction of at least 90% on Overcooked(Salad).
- Q-function trained with Qwen-1.5 7B and k=5 lifts Qwen-1.5 4B on unseen ALFWorld(Pick) tasks from 0.08 to 0.42 at k=10 in DQN-Prior tables.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Restrict offline CQL to LLM-prior actions | CQL-Prior · ALFWorld Pick | reward: 0.03 -> 0.80 | Helped |
| Restrict offline CQL to LLM-prior actions | CQL-Prior · Overcooked Salad (1000) | reward: 0.32 -> 1.01 | Helped |
| Restrict offline CQL to LLM-prior actions | CQL-Prior · Overcooked Salad (4000) | reward: 0.32 -> 1.19 | Helped |
| Restrict online DQN exploration and updates to LLM-prior actions | DQN-Prior · Overcooked Salad (1000) | reward: 0.14 -> 0.98 | Helped |
| Increase DQN-Prior action proposals from k=5 to k=10 (ablation) | DQN-Prior with Qwen-1.5 7B · ALFWorld Pick seen tasks | success rate: 0.62 -> 0.77 | Helped |
| Increase DQN-Prior action proposals from k=5 to k=10 (ablation) | DQN-Prior with Qwen-1.5 7B · ALFWorld Pick unseen tasks | success rate: 0.38 -> 0.42 | Helped |
| Increase CQL-Prior action proposals from k=5 to k=10 (ablation) | CQL-Prior with Qwen-1.5 7B · ALFWorld Pick seen tasks | success rate: 0.80 -> 0.81 | Helped |
| Increase CQL-Prior action proposals from k=5 to k=10 (ablation) | CQL-Prior with Qwen-1.5 7B · ALFWorld Pick unseen tasks | success rate: 0.46 -> 0.50 | Helped |
| Use mapped free-form LLM outputs rather than the distribution over admissible actions (ablation) | Qwen1.5-7B LLM prior · ALFWorld Pick and place | winning rate: 0% -> 19% | Helped |
| Add a KL constraint to GFlan using the LLM prior | GFlan-Prior · Online policy-based RL tasks | cumulative reward | Helped |
| Use k=5 rather than k=1 prior action proposals for the GFlan-Prior KL estimate (ablation) | GFlan-Prior · Online policy-based RL tasks | cumulative reward | Helped |
| Increase CQL training data from fewer than 12000 examples to around 1000 examples while using CQL-Prior | CQL-Prior versus CQL · Overcooked Salad | required samples: at least 90% | Helped |
Surprising findings
- The LLM prior alone (k=1) with Qwen-1.5 4B fails on ALFWorld(Pick), yet a Q-function trained with a stronger LLM prior enables it to succeed.
- Mapping free-form LLM output to actions (winning rate 19%) worked far better than normalized action-likelihood distributions (0%) on ALFWorld.
Limitations
- Only text-based games with finite action spaces are studied.
- Results depend on rule-based mapping from free-form LLM output to admissible actions.
- Performance is non-monotonic in k, with larger proposal sets sometimes introducing OOD pairs.
Experimental setup
Datasets: ALFWorld Overcooked Frozen Lake
Benchmarks: ALFWorld Overcooked Frozen Lake
Metrics: success rate cumulative reward sample efficiency
Baselines: DQN CQL Behavior Cloning GFlan PPO
Builds on: DQN CQL GFlan Control as Inference PPO
Closest papers by meaning
- On the Modeling Capabilities of Large Language Models for Sequential Decision Making (ICLR 2025 · similarity 0.92)
- ShiQ: Bringing back Bellman to LLMs (NeurIPS 2025 · similarity 0.91)
- LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models (NeurIPS 2025 · similarity 0.91)
- Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoning (NeurIPS 2025 · similarity 0.90)
- On Predictability of Reinforcement Learning Dynamics for Large Language Models (ICLR 2026 · similarity 0.90)
- Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Models (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.