Research map / Large language models / Reasoning and chain of thought
Risk-Sensitive Reinforcement Learning for Alleviating Exploration Dilemmas in Large Language Models
Yuhua Jiang, Jiawei Huang, Yufeng Yuan, Xin Mao, YuYue, Qianchuan Zhao, Lin Yan
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a risk-sensitive RLVR approach to address the exploration dilemma in LLM reasoning, where standard RL boosts pass@1 but suppresses solution diversity and pass@k.
Problem
Standard RLVR for LLMs gets stuck in local optima of sharply peaked pre-trained policies, improving single-solution accuracy (pass@1) while suppressing solution diversity and multi-solution performance (pass@k).
Approach
Applies risk-sensitive reinforcement learning to RLVR so that the policy explores beyond its sharply peaked initial distribution, preserving diversity while improving pass@k; the abstract does not detail the exact risk measure.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace mean-reward advantage with risk-sensitive advantage (RS-GRPO) | GRPO, Qwen2.5-Math-7B · Math average of 6 benchmarks (train deepmath103k) | pass@32: 45.3 -> 48.3 | Improved |
| replace mean-reward advantage with risk-sensitive advantage | GRPO, Qwen3-4B-Base · Math average of 6 benchmarks (train math12k) | pass@32: 48.1 -> 53.1 | Improved |
| replace mean-reward advantage with risk-sensitive advantage | GRPO, Qwen2.5-Math-7B · Math average of 6 benchmarks (train deepmath103k) | pass@1: 26.6 -> 28.6 | Improved |
| replace mean-reward advantage with risk-sensitive advantage | GRPO, Qwen2.5-Math-1.5B · Math average of 6 benchmarks (train deepmath103k) | pass@1: 21.4 -> 21.3 | No clear effect |
| replace mean-reward advantage with risk-sensitive advantage | GRPO, Llama-3.1-8B-Instruct · Math average of 6 benchmarks (train math12k) | pass@32: 21.0 -> 23.5 | Improved |
| apply RS-GRPO vs GRPO relative to base model at high k | RS-GRPO · Qwen2.5-7B and Llama3.1-8B-Instruct pass@k | pass@k (k>256) | Mixed |
| increase risk-sensitivity beta from 0 to 2-8 (ablation) | RS-GRPO, Qwen2.5-Math-1.5B/7B · math test benchmarks | pass@32: approx. +5% | Improved |
| use beta = 2 rather than beta = 0 (ablation) | RS-GRPO, Qwen2.5-Math-7B · math test benchmarks | pass@1: +1-2% | Improved |
| increase beta (ablation) | RS-GRPO · training set | training reward | Hurt |
| use risk-sensitive advantage instead of pass@k advantages (Chen, Mahdavi) | RS-GRPO · training dynamics Qwen2.5-Math-1.5B/7B | pass@1 | Improved |
| Walder & Karkhanis strictly positive advantages (ablation) | pass@k baseline · Math average of 6 benchmarks (train deepmath103k) | pass@1 and pass@32: 16.7 / 36.6 vs GRPO 21.4 / 37.5 | Hurt |
| train with RS-GRPO instead of GRPO on out-of-domain science QA | GRPO, Qwen2.5-Math-7B · GPQA-Diamond | pass@32: 84.8 -> 88.9 | Improved |
Experimental setup
Metrics: pass@1 pass@k
Builds on: RLVR
Closest papers by meaning
- Learning to Explore with Parameter-Space Noise: A Deep Dive into Parameter-Space Noise for Reinforcement Learning with Verifiable Rewards (NeurIPS 2026 · similarity 0.91)
- Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward (ICLR 2026 · similarity 0.89)
- Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewards (ICML 2026 · similarity 0.89)
- DeepSearch: Overcome the Bottleneck of Reinforcement Learning with Verifiable Rewards via Monte Carlo Tree Search (ICLR 2026 · similarity 0.89)
- Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement (ICML 2026 · similarity 0.89)
- Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
