AI Paper Atlas

Research map / Large language models / Reasoning and chain of thought

Risk-Sensitive Reinforcement Learning for Alleviating Exploration Dilemmas in Large Language Models

Yuhua Jiang, Jiawei Huang, Yufeng Yuan, Xin Mao, YuYue, Qianchuan Zhao, Lin Yan

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a risk-sensitive RLVR approach to address the exploration dilemma in LLM reasoning, where standard RL boosts pass@1 but suppresses solution diversity and pass@k.

Problem

Standard RLVR for LLMs gets stuck in local optima of sharply peaked pre-trained policies, improving single-solution accuracy (pass@1) while suppressing solution diversity and multi-solution performance (pass@k).

Approach

Applies risk-sensitive reinforcement learning to RLVR so that the policy explores beyond its sharply peaked initial distribution, preserving diversity while improving pass@k; the abstract does not detail the exact risk measure.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace mean-reward advantage with risk-sensitive advantage (RS-GRPO)GRPO, Qwen2.5-Math-7B · Math average of 6 benchmarks (train deepmath103k)pass@32: 45.3 -> 48.3Improved
replace mean-reward advantage with risk-sensitive advantageGRPO, Qwen3-4B-Base · Math average of 6 benchmarks (train math12k)pass@32: 48.1 -> 53.1Improved
replace mean-reward advantage with risk-sensitive advantageGRPO, Qwen2.5-Math-7B · Math average of 6 benchmarks (train deepmath103k)pass@1: 26.6 -> 28.6Improved
replace mean-reward advantage with risk-sensitive advantageGRPO, Qwen2.5-Math-1.5B · Math average of 6 benchmarks (train deepmath103k)pass@1: 21.4 -> 21.3No clear effect
replace mean-reward advantage with risk-sensitive advantageGRPO, Llama-3.1-8B-Instruct · Math average of 6 benchmarks (train math12k)pass@32: 21.0 -> 23.5Improved
apply RS-GRPO vs GRPO relative to base model at high kRS-GRPO · Qwen2.5-7B and Llama3.1-8B-Instruct pass@kpass@k (k>256)Mixed
increase risk-sensitivity beta from 0 to 2-8 (ablation)RS-GRPO, Qwen2.5-Math-1.5B/7B · math test benchmarkspass@32: approx. +5%Improved
use beta = 2 rather than beta = 0 (ablation)RS-GRPO, Qwen2.5-Math-7B · math test benchmarkspass@1: +1-2%Improved
increase beta (ablation)RS-GRPO · training settraining rewardHurt
use risk-sensitive advantage instead of pass@k advantages (Chen, Mahdavi)RS-GRPO · training dynamics Qwen2.5-Math-1.5B/7Bpass@1Improved
Walder & Karkhanis strictly positive advantages (ablation)pass@k baseline · Math average of 6 benchmarks (train deepmath103k)pass@1 and pass@32: 16.7 / 36.6 vs GRPO 21.4 / 37.5Hurt
train with RS-GRPO instead of GRPO on out-of-domain science QAGRPO, Qwen2.5-Math-7B · GPQA-Diamondpass@32: 84.8 -> 88.9Improved

Experimental setup

Metrics: pass@1 pass@k

Builds on: RLVR

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy