Research map / Reinforcement learning / RL from feedback

Follow the Winners: Conservative Policy Improvement with the Cross-Entropy Method for Critic-Free RFT

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: GRPO stability and variants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Follow the Winners (FTW), a critic-free RFT algorithm that adapts the cross-entropy method with minibatch elite (top-K of C) filtering over a replay buffer, and analyzes its risk profile against GRPO and DPO under a control-as-inference framework.

Problem

Critic-free reinforcement fine-tuning of agentic LLMs (GRPO/DPO families) updates too aggressively on high-variance, sparsely verified trajectories, and GRPO's group rollouts are impractical in stateful environments.

Approach

Replace group rollouts and critics with an ordinal top-K-of-C minibatch filter over a replay buffer, giving polynomial (not exponential) concentration on order statistics of returns; within a control-as-inference view, GRPO is risk-neutral while DPO and FTW share a bounded risk-seeking offset that FTW controls via K/C.

Key results

• On Sokoban, FTW with K in {1,2} (C=5) learns at a speed and performance comparable to GRPO and PPO, while REINFORCE++ shows large seed variance and slower learning.
• On Search-R1, FTW (C=4, K=1) matches within noise the published Search-R1 PPO and GRPO baselines on exact-match accuracy.
• For K=3 (C=5) FTW on Sokoban peaks near 0.4 success and then collapses, attributed to zero-return elites under binary rewards.

Surprising findings

• With binary Sokoban rewards, the mild filter K=3 collapses after peaking around 0.4 success rate, because minibatches with fewer than K successes fill elite slots with zero-return trajectories.
• Exponential replay-buffer weighting (FTW-exp) collapses onto the distractor arm fastest in bandit tests, and DPO is distracted earlier by noise than FTW.

Limitations

• Risk offset biases the fixed point away from the mean-optimal policy under aleatoric noise, making FTW and DPO less suited to noisy environments.
• Replay buffer adds memory overhead.
• IIA / total-ordering assumption may not hold for non-transitive settings.

Experimental setup

Datasets: Sokoban Search-R1 dataset Brax HalfCheetah

Benchmarks: Sokoban Search-R1 test set Brax HalfCheetah

Metrics: test success rate exact-match accuracy mean return L1 distance to optimal action distribution

Baselines: GRPO PPO REINFORCE++ SAC Search-R1 PPO Search-R1 GRPO

Builds on: Control-as-Inference Cross-entropy method GRPO DPO MPO

Closest papers by meaning

• Vanishing Gradients in Reinforcement Finetuning of Language Models (ICLR 2024 · similarity 0.89)
• Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards (NeurIPS 2025 · similarity 0.88)
• Agentic Reinforced Policy Optimization (ICLR 2026 · similarity 0.88)
• Learning Agentic Policy from Action Guidance (NeurIPS 2026 · similarity 0.88)
• SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Training (NeurIPS 2026 · similarity 0.87)
• Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections (NeurIPS 2025 · similarity 0.87)