Research map / Large language models / Reasoning and chain of thought

WS-GRPO: Weakly-Supervised Group-Relative Policy Optimization for Rollout-Efficient Reasoning

ICML 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes WS-GRPO, which trains a FLAN-T5-based preference model from outcome-only correct/incorrect trajectory pairs and uses its prefix-to-prefix preference margins as dense pseudo-rewards in GRPO to shorten reasoning rollouts.

Problem

GRPO's group-relative objective can encourage overlong deliberation and overthinking, and length penalties are hard to calibrate while fine-grained step supervision is unavailable beyond final-answer correctness.

Approach

Convert terminal correctness into prefix-level continue/stop guidance by training a pairwise preference model on correct vs incorrect rollouts and scoring consecutive prefixes s_{1:t-1} vs s_{1:t} as pseudo step rewards, combined with the outcome reward in GRPO.

Key results

• On ARC with Qwen2.5-7B-Instruct, WS-GRPO reaches 0.879 Pass@1 vs 0.904 for GRPO while cutting eval length from 309.30 to 16.00 tokens.
• On GSM8K with Qwen2.5-7B-Instruct, WS-GRPO drops Pass@1 from 0.924 (GRPO) to 0.851 while reducing length from 166.97 to 79.26 tokens.
• On DeepMath with Qwen3-4B-Instruct, WS-GRPO reduces length from 188.35 to 16.29 tokens but Pass@1 falls from 0.620 to 0.532.

Surprising findings

• On CommonsenseQA and DeepMath, WS-GRPO sometimes increased length and steps relative to baselines (e.g., Qwen3-4B CommonsenseQA +204% length vs GRPO), contrary to the efficiency goal.
• Accuracy drops are sizable on some settings (e.g., DeepMath Llama-3.1-8B -9.2 points), so the efficiency-accuracy trade-off is not uniformly favorable.

Limitations

• Accuracy losses on several mathematical benchmarks (e.g., DeepMath, GSM8K) are notable, and the paper acknowledges mathematical reasoning benefits less from the approach.
• Theoretical bounds rely on strong assumptions (unbiasedness of trajectory labels as step-level quality, bounded policy classes), and the consistency proof appears to be loosely argued.
• Results cover only four benchmarks and four instruction-tuned models with a single lambda=0.1 setting reported in the main text.

Experimental setup

Datasets: ARC CommonsenseQA DeepMath GSM8K

Benchmarks: ARC CommonsenseQA DeepMath GSM8K

Metrics: Pass@1 mean completion length (tokens) average reasoning steps step-efficiency (Pass@1 / steps)

Baselines: GRPO Dr.GRPO

Builds on: GRPO Dr. GRPO Bradley-Terry preference modeling

Closest papers by meaning

• BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoning (NeurIPS 2025 · similarity 0.90)
• Multi-Task GRPO: Reliable LLM Reasoning Across Tasks (ICML 2026 · similarity 0.89)
• ROLLVERIFY: BRIDGING EFFICIENCY AND ACCURACY IN LONG-TAIL ROLLOUT REINFORCEMENT LEARNING (NeurIPS 2026 · similarity 0.89)
• Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning (ICLR 2026 · similarity 0.89)
• S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models (NeurIPS 2025 · similarity 0.89)
• Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts (NeurIPS 2025 · similarity 0.89)