Research map / Large language models / Reasoning and chain of thought

Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GFPO, which samples a larger group of responses per question during GRPO-style RL, keeps only the top-k by length or reward-per-token, and computes advantages only over those, cutting response length inflation on Phi-4-reasoning-plus while preserving accuracy.

Problem

RLVR methods such as GRPO inflate response length without corresponding accuracy gains, and loss-normalization fixes alone do not curb this length growth.

Approach

Sample a larger group G per question, filter the group to the top-k responses by a target metric (shortest length or reward/length token efficiency), and compute group-normalized advantages only over the retained set, with rejected responses receiving zero advantage. Adaptive Difficulty GFPO varies k by a t-digest estimate of question difficulty.

Key results

• Shortest 8/16 GFPO cuts GRPO's excess length by 23.8% on AIME 25 and 33.0% on AIME 24 with no statistically significant accuracy drop.
• Token Efficiency GFPO reduces excess length by 70.9% on AIME 25, 84.6% on AIME 24, 79.7% on GPQA, and 82.6% on Omni-MATH.
• Shortest 6/8 GFPO (no extra sampling) reduces length inflation by only 1.8% on AIME 25, showing that sampling more is needed for length reduction.

Surprising findings

• Reducing retained responses without increasing group size (Shortest 6/8) does not meaningfully reduce length, so training-time sampling is the key lever.
• Token Efficiency GFPO produces responses shorter than the SFT model on easy AIME 25 questions while matching GRPO accuracy.

Limitations

• Experiments are on a single base model (Phi-4-reasoning-plus) and a single training run setup with 100 RL steps.
• Token Efficiency GFPO shows higher training variance and minor non-significant accuracy drops on some benchmarks.
• Training uses higher sampling cost (more responses per question) than GRPO.

Experimental setup

Datasets: 72k math problems from a larger training corpus (Phi-4-reasoning-plus RL data) LiveCodeBench (evaluation)

Benchmarks: AIME 25 AIME 24 GPQA Omni-MATH LiveCodeBench

Metrics: pass@1 accuracy average response length excess length reduction

Baselines: GRPO (Phi-4-reasoning-plus) Phi-4-reasoning (SFT) Shortest k/G GFPO

Builds on: GRPO DAPO Dr. GRPO PPO

Closest papers by meaning

• Think or Not? Selective Reasoning via Reinforcement Learning for Vision-Language Models (NeurIPS 2025 · similarity 0.90)
• Group-in-Group Policy Optimization for LLM Agent Training (NeurIPS 2025 · similarity 0.89)
• All Roads Lead to Rome: Incentivizing Divergent Thinking in Vision-Language Models (CVPR 2026 · similarity 0.89)
• BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoning (NeurIPS 2025 · similarity 0.89)
• Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasoners (ICLR 2026 · similarity 0.89)
• Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.89)