AI Paper Atlas

Research map / Reinforcement learning / Policy gradient and actor-critic

Prompt-Driven Exploration

Sunshine Jiang, John Marangola, David Zhang, Raghuram Kowdeed, Ruiyang Luo, Nitish Dashora, Richard Li, Pulkit Agrawal, Zhang-Wei Hong

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Regularized scalable deep value learning

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes prompt-driven exploration for RL post-training of foundation models, where exploration is driven by varying prompts rather than perturbing sampled actions, to reach strategies the current policy rarely produces.

Problem

RL post-training of foundation models is bottlenecked by exploration: action-perturbation methods only move the policy locally, which fails on hard tasks and especially when starting from a weak policy with low success rates.

Approach

Instead of perturbing sampled actions, explore by changing the prompt so the policy is steered toward strategies it would not otherwise produce, enabling useful behavior to be discovered from a weak starting policy.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
explore with VLM-refined prompts (PDE) instead of action noisePPO on pi0.5 · ManiSkill all taskssuccess rate (%): 62.31 -> 73.32Improved
explore with PDE instead of action noise on hard ManiSkill tasksPPO on pi0.5 · ManiSkill hardsuccess rate (%): 51.65 -> 58.77Improved
PDE on a task with zero initial successPPO on pi0.5 · LIBERO-PRO Object task 2training success (%): 0.0 -> 81.8Improved
PDE on a task where action noise stays at zeroPPO on pi0.5 · LIBERO-PRO Object task 5training success (%): 0.0 -> 77.7Improved
PDE speeds learning when action noise eventually succeedsPPO on pi0.5 · LIBERO-PRO Object task 0training success at step 30 (%): 65.2 -> 99.0Improved
PDE versus parameter noisePPO · LIBERO-PRO evaluationmean evaluation successImproved
add RND intrinsic bonus or Robometer dense reward instead of PDEPPO · LIBERO-PRO evaluationevaluation successNo clear effect
replace discovered prompts with random VLM paraphrases (ablation)PDE · LIBERO-PRO Object (task perturbation)training successHurt
use adaptive prompt sampling instead of fixed uniform sampling (ablation)PDE · LIBERO-PROtraining successImproved
use mixed backpropagation over canonical and exploratory prompts (ablation)PDE · LIBERO-PROtraining successImproved
add PDE to PPO on real robotPPO on pi0.5 · Real-world Franka, 3 tasksavg success rate (%): 20.0 -> 45.0Improved
add PDE to RLOO on LiveCodeBenchRLOO · LiveCodeBench (held-out)accuracyMixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy