Research map / Reinforcement learning / RL from feedback
Experience is the Best Teacher: Motivating Effective Exploration in Reinforcement Learning for LLMs
Wenjian Zhang, Kongcheng Zhang, Jiaxin Qi, Jianqiang Huang, Baisheng Lai
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add hindsight experience (original response plus unmet rubrics) in-context, with loss masking on the experience tokens (ablation) | RLVR on Qwen2.5-7B-Instruct · IFBench | accuracy: 31.6 -> 36.7 | Improved |
| add bonus reward on top of hindsight experience (ablation) | HeRL on Qwen2.5-7B-Instruct · IFBench | accuracy: 36.7 -> 39.7 | Improved |
| add bonus reward on top of hindsight experience (ablation) | HeRL on Llama-3.2-3B-Instruct · WritingBench | score: 41.1 -> 45.4 | Improved |
| use naive hindsight experience (train on revised responses without the experience in context) instead of RLVR baseline (ablation) | RLVR on Qwen2.5-7B-Instruct · WritingBench | score: 54.8 -> 52.7 | Hurt |
| use naive hindsight experience instead of RLVR baseline (ablation) | RLVR on Qwen3-4B-Instruct-2507 · WritingBench | score: 83.9 -> 66.2 | Hurt |
| use naive hindsight experience instead of RLVR baseline (ablation) | RLVR on Llama-3.2-3B-Instruct · HealthBench-500 | score: 17.8 -> 23.9 | Improved |
| use 2 hindsight-guided trajectories per group instead of 1 (ablation) | HeRL on Qwen2.5-7B-Instruct · IFBench | accuracy: 39.7 -> 38.5 | Hurt |
| HeRL training instead of RLVR | Qwen2.5-7B-Instruct · MulDimIF | accuracy: 73.5 -> 83.4 | Improved |
| HeRL training instead of RLVR, evaluated on out-of-distribution benchmarks | Llama-3.2-3B-Instruct · MATH-500 | accuracy: 39.6 -> 36.6 | Hurt |
| HeRL training instead of RLVR, evaluated on out-of-distribution benchmarks | Qwen3-4B-Instruct-2507 · MATH-500 | accuracy: 88.0 -> 89.8 | Improved |
| guided sampling by hindsight experience instead of stochastic sampling or entropy-based search | Qwen2.5-7B-Instruct · HealthBench (500 questions) | PassRate: 28.4 -> 38.2 at 2 attempts (guided vs first-attempt level) | Improved |
| entropy-based token branching search instead of stochastic sampling (ablation) | Qwen2.5-7B-Instruct · HealthBench (500 questions) | PassRate | Hurt |
Closest papers by meaning
- Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for Open-Ended LLM Reasoning (ICML 2026 · similarity 0.91)
- LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models (NeurIPS 2025 · similarity 0.90)
- Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward (ICLR 2026 · similarity 0.90)
- Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs (ICLR 2026 · similarity 0.90)
- EVOLvE: Evaluating and Optimizing LLMs For In-Context Exploration (ICML 2025 · similarity 0.89)
- Can large language models explore in-context? (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
