Research map / Large language models / Reasoning and chain of thought

Whatever Remains Must Be True: Filtering Drives Reasoning in LLMs, Shaping Diversity

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DMVR, which approximates an explicit verifier-filtered target distribution with alpha-DPG (f-divergence policy gradient with alpha-divergences), interpolating between forward and reverse KL to trade off precision and coverage on Lean theorem proving.

Problem

RLVR (GRPO/PPO) implicitly optimizes reverse KL to a verifier-filtered target, which is mode-seeking and causes loss of output diversity; the precision-coverage trade-off in LLM reasoning is poorly controlled.

Approach

Define the target as the base model distribution filtered by the verifier (p ∝ π_base·v) and fit the policy to it with f-DPG using the alpha-divergence family, where alpha interpolates from forward KL (coverage, RS-FT/KL-DPG) to reverse KL (precision, RLVR), with pseudo-reward clipping for stability.

Key results

• alpha-DPG with low alpha (0.25) achieves the highest pass@256 (88.5%) among compared methods, while alpha=0.999 reaches pass@1 of about 71.7% with pass@256 of 81.5%.
• alpha-DPG models lie along a precision-coverage Pareto frontier, with alpha controlling the trade-off.
• Higher tactic and premise diversity correlates with pass@256 and anticorrelates with pass@1.

Surprising findings

• Base SFT model attains broad coverage, and GRPO collapses to 16 identical sequences on one problem, while sampled responses remain highly probable under the base model.

Limitations

• Without clipping, alpha-DPG is unstable for small alpha (<=0.5).
• Evaluated only on Lean theorem proving with one base model; generalization to code or other tasks is untested.

Experimental setup

Datasets: Lean Workbook

Benchmarks: Lean Workbook test split (200 problems)

Metrics: pass@1 pass@256 Shannon index Simpson index

Baselines: GRPO Dr. GRPO High-KL GRPO-Rw-Ulkly Pass@k training GPG ReMax RLOO

Builds on: f-DPG KL-DPG GRPO Dr. GRPO RS-FT

Closest papers by meaning

• SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoning (ICML 2026 · similarity 0.92)
• Diversity-Aware Policy Optimization for Large Language Model Reasoning (NeurIPS 2025 · similarity 0.91)
• R-Diverse: Mitigating Diversity Illusion in Self-Play LLM Training (ICML 2026 · similarity 0.90)
• Diversity-Incentivized Exploration for Versatile Reasoning (ICLR 2026 · similarity 0.90)
• Diversity-Enhanced Reasoning for Subjective Questions (ICLR 2026 · similarity 0.90)
• SAGE: Shaping Anchors for Guided Exploration in RLVR of LLMs (ICML 2026 · similarity 0.89)