Research map / Reinforcement learning / RL from feedback

TIC-GRPO: Provable and Efficient Optimization for Reinforcement Learning from Human Feedback

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: GRPO stability and variants

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TIC-GRPO, a critic-free GRPO variant combining up-only importance-ratio capping with full-trajectory importance correction, with variance and convergence bounds showing a sqrt(T) horizon improvement over token-level GRPO2 and small gains on Qwen3 reasoning/coding benchmarks.

Problem

GRPO's token-level importance sampling with clipping leaves uncontrolled upper-tail importance weights on negative-advantage samples, creating a variance bottleneck that grows with response length.

Approach

Replace token-level importance ratios with a full-trajectory likelihood ratio, apply an up-only cap to it regardless of advantage sign, and normalize by the maximum response length T, so the score second moment transfers to the current policy via martingale cancellation.

Key results

• On Qwen3-1.7B, TIC-GRPO scores 11.77 on AIME24 (Avg@32) vs 9.17 for GRPO and 10.31 for GSPO.
• On Qwen3-8B, TIC-GRPO scores 33.34 on AIME24 vs 31.35 for GRPO and 30.21 for GSPO.
• Squared parameter-update norm on Qwen3-1.7B is about 26% lower mean and 76% lower temporal std than GSPO over 200 steps.

Surprising findings

• GSPO underperforms the GRPO baseline on Qwen3-8B (AIME24 30.21 vs 31.35).

Limitations

• Theoretical guarantees use maximum-length normalization and fresh conditional sampling, while experiments use sampled-token normalization and trajectory reuse, a mismatch.
• Improvements on benchmarks are modest in absolute terms and from single runs with no reported variance across seeds.
• Experiments are limited to Qwen3 models and four benchmarks.

Experimental setup

Datasets: DAPO-Math-17K AIME 1983-2022 problems

Benchmarks: AIME24 AIME25 MATH500 LiveCodeBench

Metrics: Avg@32 Avg@1

Baselines: GRPO GSPO DAPO-style GRPO GRPO+UB_Clip GRPO+Traj_IS

Builds on: GRPO PPO DAPO GSPO GRPO2

Closest papers by meaning

• PSPO: Prompt-Level Prioritization and Experience-Weighted Smoothing for Efficient Policy Optimization (AAAI 2026 · similarity 0.91)
• Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friends (ICLR 2026 · similarity 0.91)
• Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clipping (ICLR 2026 · similarity 0.90)
• Stratified GRPO: Handling Structural Heterogeneity in Reinforcement Learning of LLM Search Agents (ICML 2026 · similarity 0.89)
• GVPO: Group Variance Policy Optimization for Large Language Model Post-Training (NeurIPS 2025 · similarity 0.89)
• Learn from the Gap: Differential-Aware Advantage Pruning with Adaptive Rollout Sampling for GRPO (NeurIPS 2026 · similarity 0.89)