Research map / Reinforcement learning / RL from feedback
LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LLM-Explorer, a plug-in module where LLMs analyze an RL agent's recent action-reward trajectory and periodically output an action-exploration distribution that replaces the default epsilon-greedy or Gaussian exploration in DQN, DDPG, and related algorithms.
Problem
Existing RL exploration methods (epsilon-greedy, Gaussian noise) use preset stochastic processes that ignore task characteristics and only decay variance over training, so they cannot adapt to the agent's real-time learning status.
Approach
Use an LLM, given a textual task description and sampled action-reward trajectories (no states), to summarize learning status and output a task-specific exploration distribution (discrete) or per-dimension action biases (continuous) that is refreshed every K episodes, in a two-LLM summarize-then-generate pipeline.
Key results
- On 15 Atari games, LLM-Explorer with DQN improves total-mean human-normalized score from 14.27% to 19.59% (37.27% relative) and median by 13.84%, improving 13 of 15 games.
- Across DQN, Double-Dueling DQN, Rainbow, CURL, and TD3 on three environments each, LLM-Explorer consistently improves performance according to the paper.
- Average API cost is about $1.30 and about 10 hours wall-clock per 500k-step training run using GPT-4o mini.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace preset DQN exploration with LLM-driven adaptive exploration | DQN · Atari (15-game mean and median) | human-normalized score: 14.27 -> 19.59; 3.18 -> 3.62 | Helped |
| Replace preset DQN exploration with LLM-driven adaptive exploration | DQN · Qbert and Seaquest | human-normalized score: Qbert: 1.07 -> 1.04; Seaquest: 3.18 -> 3.05 | Hurt |
| Remove the summarize-and-suggestion stage and prompt one LLM directly (ablation) | DQN+LLM-Explorer · Alien, Freeway, MsPacman | score: Alien: 0.59 -> 0.51; Freeway: 69.71 -> 68.97; MsPacman: 2.75 -> 2.32 | Hurt |
| Remove the task description and provide only the environment name (ablation) | DQN+LLM-Explorer · Alien, Freeway, MsPacman | score: Alien: 0.59 -> 0.38; Freeway: 69.71 -> 61.26; MsPacman: 2.75 -> 1.89 | Hurt |
| Reduce action sampling density from M=100 to M=50 (ablation) | DQN+LLM-Explorer, K=1 · Alien, Freeway, MsPacman | score: Alien: 0.59 -> 0.51; Freeway: 69.71 -> 64.72; MsPacman: 2.75 -> 2.22 | Hurt |
| Increase exploration adjusting interval from K=1 to K=2 (ablation) | DQN+LLM-Explorer, M=100 · Alien, Freeway, MsPacman | score: Alien: 0.59 -> 0.38; Freeway: 69.71 -> 66.52; MsPacman: 2.75 -> 2.07 | Hurt |
| Increase action sampling density from M=100 to M=200 (ablation) | DQN+LLM-Explorer, K=1 · Alien, Freeway, MsPacman | score: Alien: 0.59 -> 0.83; Freeway: 69.71 -> 66.52; MsPacman: 2.75 -> 2.24 | Mixed |
| Use KL clipping with uniform fallback for overly skewed distributions (ablation) | DQN+LLM-Explorer · Qbert and Seaquest | score: Qbert: 301.97 -> 312.43; Seaquest: 196.15 -> 209.37 | Helped |
| Use KL clipping and reuse the last valid exploration strategy (ablation) | DQN+LLM-Explorer · Qbert and Seaquest | score: Qbert: 301.97 -> 316.92; Seaquest: 196.15 -> 221.81 | Helped |
| Use the automatically generated task description instead of the manual one (ablation) | DQN+LLM-Explorer · Atari (Average) | score: 1919.42 -> 1910.51 | No clear effect |
Surprising findings
- Models with 7B parameters or fewer (Qwen2.5-7B and 3B) generally fail to improve over raw DQN, while the 32B model performs best.
- Larger sampling density M=200 does not consistently improve over M=100, suggesting extra input can complicate LLM analysis.
Limitations
- Not universally effective; performance is limited on dynamic games such as Qbert and Seaquest where LLM outputs can be skewed.
- Depends on capable large LLMs; small open models (<=7B) fail, and API cost and latency add overhead.
- Relies on hand-written task descriptions; automatic descriptions only partly match manual performance.
Experimental setup
Datasets: Atari MuJoCo
Benchmarks: Atari MuJoCo
Metrics: human-normalized score mean human-norm improvement median human-norm improvement learning curve return token cost wall-clock time
Baselines: epsilon-greedy DQN DDPG with Gaussian noise NoisyNet Random network distillation Double-Dueling DQN Rainbow CURL TD3
Builds on: DQN DDPG TD3 Rainbow NoisyNet
Closest papers by meaning
- Expanding LLM Agent Boundaries with Strategy-Guided Exploration (NeurIPS 2026 · similarity 0.91)
- Prompted Policy Search: Reinforcement Learning through Linguistic and Numerical Reasoning in LLMs (NeurIPS 2025 · similarity 0.91)
- Efficient Reinforcement Learning with Large Language Model Priors (ICLR 2025 · similarity 0.91)
- From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systems (ICML 2024 · similarity 0.90)
- Learning Agentic Policy from Action Guidance (NeurIPS 2026 · similarity 0.90)
- EVOLvE: Evaluating and Optimizing LLMs For In-Context Exploration (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.