Research map / Reinforcement learning / RL from feedback

$f$-GRPO & Beyond: Divergence-Based Reinforcement Learning Algorithms for General LLM Alignment

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: Reward modeling versus direct optimization

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Extends divergence-estimation view of preference alignment to RLVR by defining reward-induced aligned/unaligned distributions, proposing on-policy f-GRPO and hybrid f-HAL, with theory on reward improvement and experiments on math reasoning and safety alignment.

Problem

Preference alignment objectives have been interpreted as f-divergence estimators, but this view does not cover RLVR where only scalar rewards (no chosen/rejected pairs) are observed; on-policy reward learning also suffers reward hacking with learned reward models.

Approach

Use above- vs below-average reward responses to induce aligned/unaligned distributions, plug reward-induced importance weights into the f-divergence variational (FDO) objective to get on-policy f-GRPO, and mix with off-policy preference loss (f-HAL) to mitigate reward hacking.

Key results

• On Qwen2.5-Math-1.5B trained on LIMR, Pearson-divergence f-GRPO reaches Relative Overall 86.49 and Avg Rank 2.6 vs GRPO's 74.26 and 5.2.
• On Qwen2.5-7B safety alignment, the hybrid f-HAL (RKL, lambda=0.5) reaches Total Score 66.30 vs GRPO's 38.19.
• f-GRPO (lambda=0) improves over GRPO on most divergences in safety alignment, e.g., chi^2 Total 54.62 vs 38.19 for Qwen-7B; on Qwen-1.5B, GRPO total is 16.77 vs 34.99 for Hellinger f-GRPO.

Surprising findings

• For Qwen2.5-1.5B safety alignment, direct aligners remain slightly stronger overall than hybrid f-HAL, while hybrid f-HAL still substantially improves over pure on-policy RL.
• Pure on-policy GRPO performs very poorly in safety alignment, consistent with reward hacking into blanket refusals.

Limitations

• Experiments limited to models up to 7B parameters due to resource constraints.
• Theory shows all canonical f-divergences achieve monotone reward improvement, but no single best divergence is identified; finite-sample dynamics may differ.
• Gains on math are mixed across divergences and benchmarks (e.g., AIME25 and AMC23 sometimes worse than GRPO).

Experimental setup

Datasets: GSM8K LIMR OpenRS compliance-refusal safety dataset (Safety_Accept_Reject) Anthropic and OpenAI helpful-harmless preference data

Benchmarks: GSM8K MATH500 AMC23 AIME24 AIME25 GCG-optimized AdvBench SALAD ToxiGen MMLU-Pro IFEval MuSR

Metrics: Pass@1 Relative Overall score Avg rank Bhattacharyya distance separation attack success rate accuracy AvgRob AvgUtil Total score

Baselines: GRPO DPO KTO BCO f-GRPO variants (Hellinger, JS, KL, Pearson chi^2, reverse KL, TV)

Builds on: GRPO DPO f-divergence direct alignment (FDO) KTO BCO

Closest papers by meaning

• Reinforcement Learning with Verifiable Rewards: GRPO's Loss, Dynamics, and Success Amplification (ICML 2026 · similarity 0.91)
• As Simple as Fine-tuning: LLM Alignment via Bidirectional Negative Feedback Loss (ICLR 2025 · similarity 0.91)
• AlphaPO: Reward Shape Matters for LLM Alignment (ICML 2025 · similarity 0.90)
• Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints (ICLR 2024 · similarity 0.90)
• Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms (NeurIPS 2024 · similarity 0.89)
• On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization (NeurIPS 2025 · similarity 0.88)