Research map / Large language models / Alignment and preferences
Transfer Q-star : Principled Decoding for LLM Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Transfer Q* (TQ*), a decoding-time alignment method that estimates the optimal token-level Q-function for a target reward via a DPO-aligned baseline model (direct transfer) or a baseline aligned to a different reward (indirect transfer via importance weighting), with theoretical suboptimality and KL bounds.
Problem
Decoding-time alignment requires the optimal Q-function Q*, which is unavailable; prior methods approximate it with the SFT model's Q, causing suboptimal decoding.
Approach
Estimate Q* implicitly using a trajectory-level baseline model aligned to a target reward (or a different baseline reward, corrected by an importance ratio), and use it to reweight token-level SFT/DPO probabilities at decoding time without updating weights.
Key results
- TQ* achieves up to 1.45x improvement in average reward over CD and 67.34% GPT-4 win-tie rate against CD-- on Evaluation-3 setup.
- GPT-4 win-tie rates vs ARGS-SFT 86.67% (Evaluation-1), 72.00% (Evaluation-2), 75.34% (Evaluation-3).
- Theory: suboptimality gap bounded by beta*KL(rho*||rho_SFT) minus a nonnegative decoding term; KL to SFT bounded by (1/beta + T/alpha) Rmax.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use TQ* instead of ARGS-SFT | GPT-4 response evaluation · Evaluation-1 | win-tie rate (%): 86.67 | Helped |
| use TQ* instead of ARGS-SFT | GPT-4 response evaluation · Evaluation-2 | win-tie rate (%): 72.00 | Helped |
| use TQ* instead of ARGS-SFT | GPT-4 response evaluation · Evaluation-3 | win-tie rate (%): 75.34 | Helped |
| use TQ* instead of DPO | GPT-4 response evaluation · Evaluation-1 | win-tie rate (%): 70.67 | Helped |
| use TQ* instead of DPO | GPT-4 response evaluation · Evaluation-2 | win-tie rate (%): 77.34 | Helped |
| use TQ* instead of DPO | GPT-4 response evaluation · Evaluation-3 | win-tie rate (%): 70.00 | Helped |
| use TQ* instead of ARGS-DPO | GPT-4 response evaluation · Evaluation-1 | win-tie rate (%): 68.00 | Helped |
| use TQ* instead of ARGS-DPO | GPT-4 response evaluation · Evaluation-2 | win-tie rate (%): 71.33 | Helped |
| use TQ* instead of ARGS-DPO | GPT-4 response evaluation · Evaluation-3 | win-tie rate (%): 74.00 | Helped |
| use TQ* instead of CD-- | GPT-4 response evaluation · Evaluation-1 | win-tie rate (%): 66.67 | Helped |
| use TQ* instead of CD-- | GPT-4 response evaluation · Evaluation-2 | win-tie rate (%): 65.34 | Helped |
| use TQ* instead of CD-- | GPT-4 response evaluation · Evaluation-3 | win-tie rate (%): 67.34 | Helped |
Surprising findings
- Direct transfer to the target reward (DT) performs subpar in synthetic indirect transfer tasks, while the indirect variant performs better.
Limitations
- CD baseline approximated without the adapter module due to unavailable code, so comparison is to an approximate version.
- Theoretical analysis is at population level; finite-sample guarantees are not provided.
- Decoding requires sampling k candidate tokens and rollouts of full trajectories per step, adding inference cost.
Experimental setup
Datasets: UltraFeedback HH-RLHF Berkeley Nectar
Benchmarks: UltraFeedback HH-RLHF Berkeley Nectar
Metrics: average reward win-tie rate diversity (n-gram repetition) coherence (SimCSE cosine similarity) KL divergence
Baselines: ARGS-SFT ARGS-DPO DPO CD-- Best-of-N sampling (oracle)
Builds on: DPO Controlled decoding (CD) ARGS RLHF KL-regularized objective
Closest papers by meaning
- Fast Best-of-N Decoding via Speculative Rejection (NeurIPS 2024 · similarity 0.88)
- Decoding-Time Language Model Alignment with Multiple Objectives (NeurIPS 2024 · similarity 0.87)
- Collab: Controlled Decoding using Mixture of Agents for LLM Alignment (ICLR 2025 · similarity 0.87)
- Progressive Mixed-Precision Decoding for Efficient LLM Inference (ICLR 2025 · similarity 0.86)
- Aligning Large Language Models with Representation Editing: A Control Perspective (NeurIPS 2024 · similarity 0.86)
- Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.