Research map / Large language models / Reasoning and chain of thought

The Unlearnability Phenomenon in RLVR for Language Models

ICML 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies "unlearnable" hard examples in GRPO-based RLVR that keep receiving correct rollouts yet never improve, and attributes this to flawed representations (gradient outliers, low-quality reasoning) rather than optimization factors.

Problem

Hard math training examples under RLVR (GRPO) often fail to improve even when correct rollouts are sampled; the paper asks why these 'unlearnable' examples persist despite positive reward.

Approach

Shows that unlearnability is not explained by positive-rollout scarcity, clipping, or KL regularization, but by low cross-example gradient similarity and low-quality reasoning on correct answers; data augmentation fails to fix it while mid-training raises gradient similarity.

Key results

• Across settings, after excluding examples without positive reward, roughly half of hard examples are learnable and about 17-30% are unlearnable (e.g., 30.2% for Qwen2.5-0.5B).
• Unlearnable examples are gradient outliers with much lower cosine gradient similarity to other examples than easy and learnable examples; easy examples show cosine similarity above 0.7 for Qwen.
• Oversampling with replay, clip-higher, KL removal, and SFT on correct responses do not close the learnable-unlearnable gap; mid-training (OctoThinker) raises gradient similarity on difficult examples.

Surprising findings

• Even SFT on 200 correct responses per unlearnable example and RL with rollout group size 64 on unlearnable examples alone do not improve their reward.
• Augmented similar problems that share nearly identical structure have low gradient similarity with the originals, so semantically similar examples are not similar in optimization space.

Limitations

• Experiments limited to small to mid-scale models (0.5B-3B) in the math domain.
• Unlearnability definition relies on a thresholded pass@1 criterion (tau=0.1), so boundary cases are ambiguous.
• Reasoning quality scores are produced by an LLM judge (GPT-5-mini) rather than human annotation.

Experimental setup

Datasets: MATH MATH-500 DeepScaleR

Benchmarks: MATH-500

Metrics: pass@1 pass@16 gradient cosine similarity reasoning quality score (0-5)

Baselines: GRPO with clip-higher GRPO without KL curriculum learning SFT on distilled correct responses

Builds on: GRPO DAPO DeepScaleR PPO

Closest papers by meaning

• The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.93)
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (NeurIPS 2025 · similarity 0.93)
• Reinforcement Learning for Reasoning in Large Language Models with One Training Example (NeurIPS 2025 · similarity 0.92)
• On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation (ICLR 2026 · similarity 0.92)
• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.92)
• Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.91)