Research map / Large language models / Reasoning and chain of thought

ResRL: Boosting LLM Reasoning via Negative Sample Projection Residual Reinforcement Learning

ICML 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ResRL, an RLVR method that reweights negative-sample penalties by the projection residual of negative token hidden states onto an SVD-based low-rank positive subspace, improving Avg@16 while preserving Pass@k diversity.

Problem

RLVR reduces generation diversity, and negative sample reinforcement (NSR) suppresses semantic content shared between positive and negative responses, causing gradient conflict and lazy likelihood displacement that limits Pass@1 gains.

Approach

Derives a single-forward proxy (orthogonal-complement energy of negative token representations relative to a positive SVD subspace) that upper-bounds representation alignment, and uses it to downweight negative-token penalties for tokens aligned with positive semantics.

Key results

• On math with Qwen3-4B, ResRL reaches 57.0 average Avg@16 versus 52.1 for NSR (Weighted-Reinforce) and 53.1 for GRPO.
• ResRL improves Pass@128 by 7.0% over NSR on Qwen3-4B, averaged over AIME24, AIME25 and AMC23.
• On ALFWorld with Qwen2.5-7B-Instruct, ResRL reaches 86.7% overall success versus 78.5% for EMPG and 80.4% for PPO.

Surprising findings

• Removing the KL penalty improves AIME2024 accuracy by 9% while remaining stable, suggesting the projection weighting acts as an intrinsic regularizer.
• On Qwen3-8B, ResRL underperforms NSR on AIME24 (50.8 vs 55.4) and on AMC23 is roughly tied, so gains are not uniform across benchmarks.

Limitations

• Gains are not uniform: ResRL trails on some subsets such as AIME24 for Qwen3-8B and MATH500 for Qwen3-4B relative to baselines.
• The theoretical bound relies on approximations (treating the logit-space factor as unmodeled and assuming positives are well covered by the subspace with an additive error term).
• Rank, quantile, and subspace budget hyperparameters require sweeps to tune.

Experimental setup

Datasets: DAPO training set DeepCoder dataset ToolRL training set ALFWorld WebShop

Benchmarks: AIME 2024 AIME 2025 AMC 2023 MATH500 Minerva OlympiadBench LiveCodeBench CodeForces HumanEval+ ALFWorld WebShop BFCL

Metrics: Avg@16 Pass@k Pass@16 Pass@128 CodeForces rating and percentile success rate accuracy

Baselines: GRPO DAPO FlowRL NSR PPO ReAct EMPG ResT ToolACE

Builds on: GRPO Negative Sample Reinforcement DAPO FlowRL EMPG

Closest papers by meaning

• The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning (NeurIPS 2025 · similarity 0.92)
• The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.90)
• Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.89)
• Diversity-Aware Policy Optimization for Large Language Model Reasoning (NeurIPS 2025 · similarity 0.89)
• TTRL: Test-Time Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
• Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs (ICLR 2026 · similarity 0.88)