Research map / Reinforcement learning / Policy gradient and actor-critic
Prompt-Driven Exploration
Sunshine Jiang, John Marangola, David Zhang, Raghuram Kowdeed, Ruiyang Luo, Nitish Dashora, Richard Li, Pulkit Agrawal, Zhang-Wei Hong
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes prompt-driven exploration for RL post-training of foundation models, where exploration is driven by varying prompts rather than perturbing sampled actions, to reach strategies the current policy rarely produces.
Problem
RL post-training of foundation models is bottlenecked by exploration: action-perturbation methods only move the policy locally, which fails on hard tasks and especially when starting from a weak policy with low success rates.
Approach
Instead of perturbing sampled actions, explore by changing the prompt so the policy is steered toward strategies it would not otherwise produce, enabling useful behavior to be discovered from a weak starting policy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| explore with VLM-refined prompts (PDE) instead of action noise | PPO on pi0.5 · ManiSkill all tasks | success rate (%): 62.31 -> 73.32 | Improved |
| explore with PDE instead of action noise on hard ManiSkill tasks | PPO on pi0.5 · ManiSkill hard | success rate (%): 51.65 -> 58.77 | Improved |
| PDE on a task with zero initial success | PPO on pi0.5 · LIBERO-PRO Object task 2 | training success (%): 0.0 -> 81.8 | Improved |
| PDE on a task where action noise stays at zero | PPO on pi0.5 · LIBERO-PRO Object task 5 | training success (%): 0.0 -> 77.7 | Improved |
| PDE speeds learning when action noise eventually succeeds | PPO on pi0.5 · LIBERO-PRO Object task 0 | training success at step 30 (%): 65.2 -> 99.0 | Improved |
| PDE versus parameter noise | PPO · LIBERO-PRO evaluation | mean evaluation success | Improved |
| add RND intrinsic bonus or Robometer dense reward instead of PDE | PPO · LIBERO-PRO evaluation | evaluation success | No clear effect |
| replace discovered prompts with random VLM paraphrases (ablation) | PDE · LIBERO-PRO Object (task perturbation) | training success | Hurt |
| use adaptive prompt sampling instead of fixed uniform sampling (ablation) | PDE · LIBERO-PRO | training success | Improved |
| use mixed backpropagation over canonical and exploratory prompts (ablation) | PDE · LIBERO-PRO | training success | Improved |
| add PDE to PPO on real robot | PPO on pi0.5 · Real-world Franka, 3 tasks | avg success rate (%): 20.0 -> 45.0 | Improved |
| add PDE to RLOO on LiveCodeBench | RLOO · LiveCodeBench (held-out) | accuracy | Mixed |
Closest papers by meaning
- Representation-Based Exploration for Language Models: From Test-Time to Post-Training (ICLR 2026 · similarity 0.89)
- Prompted Policy Search: Reinforcement Learning through Linguistic and Numerical Reasoning in LLMs (NeurIPS 2025 · similarity 0.88)
- Expanding LLM Agent Boundaries with Strategy-Guided Exploration (NeurIPS 2026 · similarity 0.88)
- From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systems (ICML 2024 · similarity 0.87)
- Can large language models explore in-context? (NeurIPS 2024 · similarity 0.87)
- LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
