Research map / Reinforcement learning / RL from feedback
$Q\sharp$: Provably Optimal Distributional RL for LLM Post-Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Q♯, a value-based KL-regularized RL method that guides a frozen reference LLM with a distributionally learned optimal soft Q function, trained by iterative data aggregation and MLE/BCE.
Problem
Policy-based RL post-training (PPO, DPO, REINFORCE) struggles to fix shortcuts inherited from pre-training, and existing value-guided decoding methods (CD, VAS) guide with unregularized Q-values, which does not converge to the KL-regularized optimal policy.
Approach
Guide the frozen reference policy with the optimal KL-regularized Q function, expressed as a functional of the learned cumulative-reward distribution of the reference policy; learn that distribution with distributional (MLE/BCE) supervised losses on data aggregated across iterations (DAgger-style), with no TD bootstrapping.
Key results
- On GSM8K with Llama 3.1 8B, Q♯ reaches 85.1 pass@1 vs 82.9 for the reference and 84.5 for CD, with KL 3.67 vs 7.43 for CD.
- On MATH with Llama 3.1 8B, Q♯ reaches 46.7 pass@1 vs 43.9 for the reference and 45.3 for CD, with KL 8.69 vs 26.8 for CD.
- On the star-graph task, Q♯ and CD fix the shortcut to near-perfect accuracy while REINFORCE and RPO plateau at 1/d and DPO collapses to zero accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Guide the reference model with Q♯ | Llama 3 8B · GSM8K | pass@1: 69.1 -> 78.4 | Helped |
| Guide the reference model with Q♯ rather than CD | Llama 3 8B · GSM8K | KL-Divergence: 6.39 -> 2.65 | Helped |
| Guide the reference model with Q♯ | Llama 3.1 8B · MATH-500 | pass@1: 43.9 -> 46.7 | Helped |
| Add Q♯ guidance | Llama 3.1 70B · MATH-500 | pass@1: 60.6 -> 63.1 | Helped |
| Increase Q♯ model size from 1B to 3B | Llama 3 70B · MATH-500 | pass@1: 46.4 -> 46.7 | Helped |
| Train on all later prefixes rather than only the current prefix (ablation) | Q♯ · GSM8K validation | pass@1: 80.5 -> 82.3 | Helped |
| Use V-type rather than Q-type parameterization (ablation) | Q♯ · MATH validation | pass@1: 66.4 -> 67.4 | Helped |
| Use distributional loss rather than MSE (ablation) | Q♯ · MATH validation | pass@1: 65.4 -> 67.4 | Helped |
| Use two iterations rather than one (ablation) | Q♯ · GSM8K validation | pass@1: 82.3 -> 83.5 | Helped |
| Use two iterations rather than one (ablation) | Q♯ · MATH validation | pass@1: 67.4 -> 68.5 | Helped |
Surprising findings
- DPO often pushes down both chosen and rejected paths, collapsing accuracy to zero on star-graph tasks.
- CD's KL grows rapidly as eta decreases and its performance can fall below the reference policy, whereas Q♯ stays stable.
Limitations
- Theory applies to deterministic MDPs; stochastic transitions would require TD learning with completeness assumptions.
- Inference requires extra forward passes of the value model at each decoding step, and V-type naively needs |V| passes without the top-20 approximation.
Experimental setup
Datasets: GSM8K MATH QuALITY AIME-24 star-graph synthetic graphs
Benchmarks: GSM8K MATH-500 AIME-24 QuALITY star-graph
Metrics: pass@1 maj1@8 KL divergence test accuracy
Baselines: Controlled Decoding (CD) REINFORCE DPO RPO Value augmented sampling (VAS)
Builds on: controlled decoding DAgger soft Q-learning DPO
Closest papers by meaning
- Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.90)
- Tricks or Traps? A Deep Dive into RL for LLM Reasoning (ICLR 2026 · similarity 0.90)
- SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning (NeurIPS 2026 · similarity 0.89)
- Reinforcement Learning for Reasoning in Large Language Models with One Training Example (NeurIPS 2025 · similarity 0.88)
- The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning (NeurIPS 2025 · similarity 0.88)
- ReST-RL: Reinforcing LLM Reasoning through Unified Self-Training and Value-Guided Search (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.