Research map / Reinforcement learning / RL from feedback
CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models
Runpeng Dai, Linfeng Song, Haolin Liu, Zhenwen Liang, Dian Yu, Haitao Mi, Zhaopeng Tu, Rui Liu, Tong Zheng, Hongtu Zhu, Dong Yu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add perplexity-based actor curiosity bonus | GRPO on Qwen3-4B-Base · Average over MATH/AMC23/AIME24/AIME25 | accuracy: 48.2 -> 50.6 | Improved |
| add perplexity-based actor curiosity bonus | GRPO on Qwen3-4B-Base · AIME24 | Pass@16: 41.9 -> 48.5 | Improved |
| add perplexity bonus | PPO on Qwen3-4B-Base · Average over benchmarks | accuracy: 46.5 -> 47.7 | Improved |
| use multi-head critic with 4 heads | PPO on Qwen3-4B-Base · Average over benchmarks | accuracy: 46.5 -> 48.5 | Improved |
| use multi-head critic with 2 heads (ablation) | PPO on Qwen3-4B-Base · Average over benchmarks | accuracy: 46.5 -> 46.6 | No clear effect |
| increase heads beyond 4 (ablation) | multi-head PPO · Average over benchmarks | accuracy: 48.5 (4) vs 48.1 (8) vs 48.6 (16) | No clear effect |
| no weight decay on PPL bonus versus staircase decay (ablation) | GRPO with PPL bonus · Average over benchmarks | accuracy: 50.6 (staircase) -> 48.2 (no decay) | Hurt |
| staircase instead of linear or cosine decay (ablation) | GRPO with PPL bonus · Average over benchmarks | accuracy: 49.7 (cosine) -> 50.6 (staircase) | Improved |
| sub-sample fraction zeta 0.5 vs 1 (ablation) | multi-head PPO (4 and 16 heads) · Average over benchmarks | accuracy: 48.6 vs 48.4 (16 heads); 48.4 vs 48.5 (4 heads) | No clear effect |
| add PPL bonus to GRPO | GRPO · entropy during training | policy entropy | Improved |
| add PPL bonus to GRPO | GRPO · training calibration | PPL gap between correct and incorrect | Improved |
| replace learned exploration with hash-based count bonus (SimHash) | GRPO · preliminary experiment | hash cell distribution | Hurt |
Closest papers by meaning
- Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward (ICLR 2026 · similarity 0.91)
- Representation-Based Exploration for Language Models: From Test-Time to Post-Training (ICLR 2026 · similarity 0.90)
- Learning to Explore with Parameter-Space Noise: A Deep Dive into Parameter-Space Noise for Reinforcement Learning with Verifiable Rewards (NeurIPS 2026 · similarity 0.90)
- Reasoning with Exploration: An Entropy Perspective (AAAI 2026 · similarity 0.90)
- SAGE: Shaping Anchors for Guided Exploration in RLVR of LLMs (ICML 2026 · similarity 0.90)
- Count Counts: Motivating Exploration in LLM Reasoning with Count-based Intrinsic Rewards (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
