Research map / Reinforcement learning / RL from feedback

Controlled Decoding from Language Models

ICML 2024 · Reinforcement learning › RL from feedback · approach: RL for LLM reasoning · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes controlled decoding (CD), a modular inference-time method that trains a DQN-style prefix scorer (value function) on a frozen PaLM 2-XXS base model to solve a tokenwise KL-regularized RL objective, with tokenwise and blockwise best-of-K variants.

Problem

Aligning frozen language models toward high-reward outcomes with configurable, inference-time control rather than retraining the generator, while keeping KL drift from the base model small.

Approach

Derives that the optimal KL-regularized tokenwise policy is the base LM distribution reweighted by exp(lambda * V*), learns V* with a DQN-style prefix scorer (CD-Q, off-policy Bellman target) and uses it for tokenwise sampling or blockwise best-of-K reranking of short blocks.

Key results

• Blockwise CD-Q with K=6 matches best-of-K with K=50 on the response-length vs KL tradeoff.
• On HH and summarization, blockwise CD-Q outperforms IPO and PPO in win rate vs KL but does not match best-of-K.
• CD-Q prefix scorer trained on PaLM 2-XXS transfers to PaLM 2-S and PaLM 2-XS without retraining and stays on par with best-of-K.

Measured effects reported in the paper

Surprising findings

• Best-of-K achieves better reward-KL tradeoffs than KL-regularized PPO in the length experiment.
• As a classifier, CD-Q and CD-FUDGE reach only about 0.6-0.63 accuracy on Anthropic HH versus about 0.71 for the reward model it was trained from.

Limitations

• Tokenwise control gives weaker HH improvements than blockwise methods, and neither blockwise CD variant matches best-of-K on HH or summarization.
• Prefix scorer classification accuracy on HH is weaker than the reward model, attributed to noisy training data.
• Tokenwise formulation is more restrictive than sequence-level RLHF, and CD's inference cost (one prefix scorer call per token) is nontrivial.

Experimental setup

Datasets: DSTC8 Reddit conversations corpus Anthropic HH TL;DR

Benchmarks: Reddit conversations corpus eval set (length) Anthropic HH test set TL;DR test set

Metrics: normalized expected reward win rate vs base policy KL divergence reward vs KL tradeoff classification accuracy

Baselines: KL-regularized PPO DPO IPO best-of-K CD-FUDGE blockwise CD-FUDGE

Builds on: FUDGE DQN KL-regularized PPO DPO IPO

Closest papers by meaning

• Collab: Controlled Decoding using Mixture of Agents for LLM Alignment (ICLR 2025 · similarity 0.90)
• Coupled Variational Reinforcement Learning for Language Model General Reasoning (ICML 2026 · similarity 0.89)
• MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions (ICLR 2025 · similarity 0.88)
• $\textbf{Re}^{2}$: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving (ICLR 2026 · similarity 0.88)
• Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization (ICLR 2025 · similarity 0.88)
• Reducing the Probability of Undesirable Outputs in Language Models Using Probabilistic Inference (NeurIPS 2025 · similarity 0.88)