Research map / Reinforcement learning / RL from feedback
Prompted Policy Search: Reinforcement Learning through Linguistic and Numerical Reasoning in LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Prompted Policy Search (ProPS), where an LLM (GPT-4o) directly proposes linear/tabular policy parameters in-context from episodic reward history, with an optional semantic-prompt variant (ProPS+), evaluated on 15 Gymnasium tasks against seven RL baselines.
Problem
Traditional RL relies on scalar rewards and cannot exploit rich linguistic knowledge such as task descriptions, domain knowledge, and expert hints during policy search.
Approach
Place an LLM at the center of the policy optimization loop: it reads the history of parameter-reward pairs (plus optional natural-language task descriptions and hints) and proposes new policy parameters, with its textual justifications acting as textual gradients, without any external optimizer.
Key results
- ProPS outperforms all baselines on 7 of 15 tasks (main text also claims 8 of 15 for ProPS+), e.g., Swimmer 218.83 vs 52.96 for the best baseline (TRPO).
- ProPS+ reaches top performance in 8 of 15 tasks, versus 5 for TRPO and 2 for ProPS; on Nim it rises from 0.33 to 0.97.
- Expert hints improve Walker by 62% and Cliff Walking by 43% in reward (Table of hint ablation), and GRPO-tuned Qwen2.5-14B improves over baseline on Mountain Car, Inverted Pendulum, and Pong.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add semantic context to ProPS | ProPS with GPT-4o · Across 15 Gymnasium tasks | number of tasks with top performance: 8 out of 15 tasks | Helped |
| Add semantic context to ProPS (ablation) | ProPS with GPT-4o · Frozen Lake | mean best episodic reward: 0.57 $\pm$ 0.17 -> 0.19 $\pm$ 0.05 | Hurt |
| Add human-provided hints to ProPS+ (ablation) | ProPS+ with GPT-4o · Mountain Car (C) | mean best episodic reward: 89.16 $\pm$ 29.72 -> 98.70 $\pm$ 0.89 | Helped |
| Add human-provided hints to ProPS+ (ablation) | ProPS+ with GPT-4o · Navigation | mean best episodic reward: 2779.55 $\pm$ 270.65 -> 3022.66 $\pm$ 135.54 | Helped |
| Add human-provided hints to ProPS+ (ablation) | ProPS+ with GPT-4o · Inverted Double Pendulum | mean best episodic reward: 148.39 $\pm$ 48.65 -> 161.07 $\pm$ 52.28 | Helped |
| Add human-provided hints to ProPS+ (ablation) | ProPS+ with GPT-4o · Cliff Walking | mean best episodic reward: -96.4 $\pm$ 22.90 -> -54.60 $\pm$ 1.18 | Helped |
| Add human-provided hints to ProPS+ (ablation) | ProPS+ with GPT-4o · Reacher | mean best episodic reward: -18.15 $\pm$ 22.06 -> -9.66 $\pm$ 1.33 | Helped |
| Add human-provided hints to ProPS+ (ablation) | ProPS+ with GPT-4o · Walker | mean best episodic reward: 126.75 $\pm$ 136.44 -> 205.47 $\pm$ 91.60 | Helped |
| Use full in-context parameter-reward history instead of a single example (ablation) | ProPS on Mountain Car · Mountain Car (C) | average reward: 100 -> 200 | Helped |
| Fine-tune Qwen2.5-14B with GRPO on Mountain Car policy evaluations (ablation) | Qwen2.5-14B prompted policy search · Mountain Car | average max reward: 8.11 $\pm$ 24.45 -> 42.01 $\pm$ 48.1 | Helped |
| Fine-tune Qwen2.5-14B with GRPO on Mountain Car policy evaluations (ablation) | Qwen2.5-14B prompted policy search · Inverted Pendulum | average max reward: 55.64 $\pm$ 36.5 -> 101.148 $\pm$ 94.804 | Helped |
| Fine-tune Qwen2.5-14B with GRPO on Mountain Car policy evaluations (ablation) | Qwen2.5-14B prompted policy search · Pong | average max reward: 0.52 $\pm$ 0.2 -> 0.66 $\pm$ 0.39 | Helped |
Surprising findings
- ProPS underperforms baselines on Walker and Maze, and ProPS+ with task descriptions degrades FrozenLake because the LLM assumes deterministic dynamics despite a 'slippery' description.
- LLMs are sensitive to the order in which information is presented in the prompt, with long tables at the start hindering reasoning.
Limitations
- Limited to policies with about 100 parameters; deep RL with neural network policies is not established beyond preliminary results.
- Sensitive to prompt structure and information ordering; semantic context can introduce misleading inductive biases.
- Reliance on proprietary LLMs with API costs and runtime that includes API latency.
Experimental setup
Datasets: Gymnasium environments OpenAI Gym environments randomly generated policy parameters
Benchmarks: Mountain Car Inverted Pendulum Inverted Double Pendulum Reacher Swimmer Hopper Walker FrozenLake Cliff Walking Maze Nim Navigation
Metrics: mean best episodic reward standard error
Baselines: DQN DDPG TD3 SAC TRPO PPO A2C
Builds on: OPRO TextGrad GRPO Stable-Baselines3
Closest papers by meaning
- LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models (NeurIPS 2025 · similarity 0.91)
- Synthesizing Programmatic Reinforcement Learning Policies with Large Language Model Guided Search (ICLR 2025 · similarity 0.90)
- On the Modeling Capabilities of Large Language Models for Sequential Decision Making (ICLR 2025 · similarity 0.90)
- DeepSearch: Overcome the Bottleneck of Reinforcement Learning with Verifiable Rewards via Monte Carlo Tree Search (ICLR 2026 · similarity 0.90)
- Expanding LLM Agent Boundaries with Strategy-Guided Exploration (NeurIPS 2026 · similarity 0.90)
- Reward Is Enough: LLMs Are In-Context Reinforcement Learners (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.