Research map / Large language models / Reasoning and chain of thought

Nudging the Boundaries of LLM Reasoning

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes NuRL, which during GRPO training injects self-generated abstract hints into problems where all rollouts fail, turning unsolvable samples into learnable ones and raising pass@k.

Problem

Online RL methods like GRPO cannot learn from problems the model never solves in its rollouts (0% pass rate), so these hard samples yield zero gradient and the model's pass@k upper limit stays unchanged.

Approach

Self-generated abstract hints (conditioned on the gold answer, then abstracted to avoid revealing it) are appended to questions with 0% pass rate during GRPO to produce non-zero-reward rollouts; hints are used only in training, not at test time.

Key results

• Self-hint NuRL improves average pass@1 over GRPO by +1.62 (Llama), +1.75 (OctoThinker), and +0.79 (Qwen) points on six benchmarks.
• On GPQA, pass@1024 rises from 63.6% (GRPO-level baseline reported as 63.4% for GRPO) to 69.7% with self-generated hints; on Date Understanding, pass@1024 goes from 86.4% to 94.0%.
• With GPT-o4-mini hints, Llama average improves +3.44 points over GRPO; hints that reveal the gold answer degrade performance.

Surprising findings

• Hints that directly reveal the gold answer harm generalization, often via reward hacking where the model outputs the provided answer.
• Applying hints from the start of training or indiscriminately underperforms applying them after GRPO convergence with a difficulty trigger.

Limitations

• Gains on Qwen are small (+0.79 average), possibly due to few stage-2 samples after filtering easy problems.
• Hints are used in training only and the method requires a converged GRPO stage 1 and gold answers to generate hints.

Experimental setup

Datasets: Open-R1 Mixture-of-Thought MATH 500 MATH Hard AIME 2024 GPQA Diamond MMLU-Pro Date Understanding MathArena OlymMATH

Benchmarks: MATH 500 MATH Hard AIME 2024 GPQA Diamond MMLU-Pro Date Understanding AIME BRUMO CMIMC HMMT OlymMATH

Metrics: pass@1 pass@k average accuracy

Baselines: Zero-shot Few-shot RFT ReFT GRPO

Builds on: GRPO DAPO ReFT RFT STaR verl

Closest papers by meaning

• Tricks or Traps? A Deep Dive into RL for LLM Reasoning (ICLR 2026 · similarity 0.91)
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale (NeurIPS 2025 · similarity 0.90)
• Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration (ICML 2026 · similarity 0.90)
• The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning (NeurIPS 2025 · similarity 0.90)
• Advancing LLM Reasoning with Natural Language and Numerical Feedback (ICML 2026 · similarity 0.90)
• Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts (NeurIPS 2025 · similarity 0.89)