Research map / Large language models / Agents, code and math
On the Modeling Capabilities of Large Language Models for Sequential Decision Making
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically studies LLMs for RL via direct policy generation versus indirect reward modeling (especially AI-feedback Bradley-Terry rewards), and shows fine-tuning for reward modeling preserves LLM knowledge better than fine-tuning for action output.
Problem
Whether LLMs, without or with fine-tuning, can serve as decision-making policies directly or as reward models for RL agents across diverse interactive domains, and how fine-tuning affects their prior knowledge.
Approach
Compares direct policy modeling (LLM outputs actions) with indirect modeling where the LLM provides AI-feedback preferences over observations, trained into a Bradley-Terry reward model that drives an off-the-shelf RL agent; further fine-tunes PaliGemma on GPT-4o captions to improve reward modeling while limiting catastrophic forgetting.
Key results
- AI feedback is the only indirect reward method that succeeds across all environments and modalities, while LLM Policy performs well mainly on MiniWob-Hard.
- Fine-tuning PaliGemma for AI feedback on ~100 GPT-4o-annotated image-caption pairs yields gains with only marginal drops on POPE, GQA, AI2D, and MMMU, whereas behaviour cloning causes catastrophic forgetting.
- AI feedback reward achieved an average score of 89.93 versus 86.3 for human-designed rewards across environments.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AI Feedback instead of other LLM-derived reward methods | LLM reward modeling · MiniWob-Hard, Wordle, NetHack, and MetaWorld | task performance | Helped |
| use AI Feedback reward modeling instead of direct action generation | LLM-based sequential decision making · MiniWob-Hard, Wordle, NetHack, and MetaWorld | task performance | Mixed |
| use Direct Scalar rewards instead of AI Feedback | LLM reward modeling · MiniWob-Hard | success rate | No clear effect |
| use Embedding-based rewards instead of AI Feedback | LLM reward modeling · the evaluated domains | task performance | Hurt |
| use AI Feedback rewards instead of human-designed environment rewards | RL agents · the evaluated domains | average score: 86.3 -> 89.93 | Helped |
| add expert demonstration to Reward as Code (ablation) | GPT-4o Reward as Code · MetaWorld, RGB observations | success rate: 0% ± 1% -> 79% ± 7% | Helped |
| add background functional knowledge to Reward as Code (ablation) | Llama 3 70B Reward as Code · MetaWorld, proprioceptive observations | success rate: 0% ± 1% -> 10% ± 3% | Helped |
| add background functional knowledge to Reward as Code (ablation) | GPT-4o Reward as Code · MetaWorld, proprioceptive observations | success rate: 5% ± 3% -> 76% ± 6% | Helped |
| combine chain-of-thought, in-context learning, and self-refinement prompting (ablation) | GPT-4o direct policy modeling · all domains and tasks | performance | Helped |
| fine-tune PaliGemma for AI Feedback using GPT-4o-annotated image-caption pairs | PaliGemma reward modeling · MetaWorld, sweep-into | success rate | Helped |
| fine-tune for AI Feedback rather than direct action selection | PaliGemma · POPE, GQA, AI2D, and MMMU | multimodal reasoning benchmark performance | Helped |
| increase task-specific fine-tuning epochs for image-caption prediction (ablation) | PaliGemma · AI2D | benchmark performance | Helped |
Surprising findings
- Performance on AI2D improves as the number of task-specific fine-tuning epochs increases.
- Direct LLM policies underperform even with GPT-4o and extensive prompting, with the LLM showing near-chance accuracy at probing action and dynamics knowledge in most domains.
Limitations
- Offline preference elicitation led to performance collapse in NetHack; Markovian reward assumption limits complex tasks.
- Direct policy modeling is limited by LLMs' poor zero-shot understanding of action spaces and transition dynamics.
- Reward as Code effectiveness depends on access to symbolic features or functional knowledge.
Experimental setup
Datasets: MiniWob NetHack Wordle MetaWorld POPE GQA AI2D MMMU
Benchmarks: MiniWob-Hard NetHack Wordle MetaWorld POPE GQA AI2D MMMU
Metrics: success rate normalized in-game score perplexity accuracy
Baselines: LLM Policy Direct Scalar Reward as Code Embedding-based reward environment reward behaviour cloning fine-tuning
Builds on: Reward as Code Recursive Criticism and Improvement Bradley-Terry model RT-2 VQ-VAE
Closest papers by meaning
- Efficient Reinforcement Learning with Large Language Model Priors (ICLR 2025 · similarity 0.92)
- Can large language models explore in-context? (NeurIPS 2024 · similarity 0.91)
- Progress Reward Model for Reinforcement Learning via Large Language Models (NeurIPS 2025 · similarity 0.91)
- Latent Reward: LLM-Empowered Credit Assignment in Episodic Reinforcement Learning (AAAI 2025 · similarity 0.90)
- Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Making (ICML 2025 · similarity 0.90)
- Scaling Autonomous Agents via Automatic Reward Modeling And Planning (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.