Research map / Reinforcement learning / RL from feedback

SALT: When More Rollouts Don’t Help in Group-Based Policy Optimization and How to Make Them Matter

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: GRPO stability and variants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SALT, a plug-in that reweights GRPO-style group-relative advantages using a mini-batch Gram subspace (LM-head gradient proxy) to reduce signed cancellation, improving math RLVR accuracy on DeepSeek-Distill-Qwen models.

Problem

Increasing the number of rollouts per prompt in GRPO-style RLVR yields diminishing returns because per-sample policy-gradient features concentrate in a low-rank, signed geometry that cancels during group-relative aggregation.

Approach

Estimate a dominant shared subspace from the mini-batch Gram matrix of LM-head gradient proxies, split advantages into a shared (projected) channel and a residual positive-part channel, and amplify the residual channel by an adaptive mixing coefficient 1 - n_eff when cancellation is severe.

Key results

• On DeepSeek-Distill-Qwen-1.5B with MATH-TRAIN, GRPO+SALT reaches 32.1 AIME24 ACC vs 29.3 for GRPO (62.9 vs 59.3 Pass@8).
• Averaged over tasks, SALT improves the backbone by about +2.54 points per setting, with ~7.7% average training-time overhead.
• Cross-model average gains of about +2.5 to +4.0 ACC (e.g., Qwen-2.5-Math-1.5B 43.7 to 46.9; Deepseek-Math-7B 33.2 to 37.2).

Surprising findings

• Entropy-increasing controls (entropy bonus, clip-cov, KL-cov) raise entropy but do not consistently improve PR, n_eff, or AIME25 performance.

Limitations

• Uses an LM-head gradient proxy whose fidelity may vary across architectures and training regimes.
• Biased geometry-reweighted surrogate; gains depend on cancellation reduction offsetting coefficient distortion.
• Evaluated only on math and code RLVR with DeepSeek-distilled and a few other models; noisier learned rewards and agentic tasks not tested.

Experimental setup

Datasets: MATH-Train-7.5k DAPO-Math-17k MBPP

Benchmarks: AIME24 AIME25 GSM8K MATH-500 GPQA-Diamond HumanEval HumanEval+

Metrics: accuracy Pass@8 participation ratio effective sample size

Baselines: GRPO DAPO Entropy Bonus Clip-Cov KL-Cov

Builds on: GRPO DAPO PPO VeRL

Closest papers by meaning

• Learn from the Gap: Differential-Aware Advantage Pruning with Adaptive Rollout Sampling for GRPO (NeurIPS 2026 · similarity 0.88)
• Leveraging Error Diversity in Group Rollouts for Reinforcement Learning (NeurIPS 2026 · similarity 0.88)
• Listwise Policy Optimization: Group-based RLVR as Target-Projection on the LLM Response Simplex (NeurIPS 2026 · similarity 0.88)
• Group-in-Group Policy Optimization for LLM Agent Training (NeurIPS 2025 · similarity 0.87)
• Rollout Pass-Rate Control: Steering Binary-Reward RL Toward Its Most Informative Regime (NeurIPS 2026 · similarity 0.87)
• ROLLVERIFY: BRIDGING EFFICIENCY AND ACCURACY IN LONG-TAIL ROLLOUT REINFORCEMENT LEARNING (NeurIPS 2026 · similarity 0.87)