Research map / Reinforcement learning / RL from feedback
ShiQ: Bringing back Bellman to LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives a Bellman-based, off-policy, token-level Q-learning loss (ShiQ) that reads LLM logits as shifted Q-values so the softmax of the logits is the regularized optimal policy, and evaluates it on toy tasks, HH/UltraFeedback, and BFCL-V3.
Problem
Q-learning methods have been little used for RL fine-tuning of LLMs; naive application of soft Q-learning to LLM logits is inefficient because of memory cost of extra networks, poor reference-logit initialization, and sparse sequence-level rewards.
Approach
Reparameterize the Bellman equation of a KL-regularized finite-horizon MDP so that LLM logits directly give the optimal policy via softmax, add a reference-shifted value term so that reference logits are a natural initialization (zero loss when r=0), and add a multi-step path-consistency extension, yielding a single-network, off-policy, token-level loss.
Key results
- On the HH dataset, ShiQ and CoPG achieve comparable reward in the offline setting, while DPO and DRO show a more limited reward range.
- On the toy grid-world with fine-grained rewards, only ShiQ discovers the intermediate reward and drives regret close to zero, whereas DPO and CoPG fail.
- On BFCL-V3 multi-turn function calling, ShiQ outperforms multi-turn DPO and CoPG in cumulative per-turn reward.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ShiQ instead of DPO in the offline bandit setting | 3-armed bandit · Synthetic offline bandit | convergence to optimal policy | Helped |
| Use ShiQ instead of DPO or CoPG with terminal-reward trajectories | 5×5 grid-world MDP · Fine-grained rewards | regret: regret 4; close to zero | Helped |
| Use ShiQ instead of the multi-turn DPO variant or CoPG | R7B · BFCL-V3 validation | verifiable reward | Helped |
| Use ShiQ instead of DPO or DRO | Cohere R7B · Anthropic-Harmless and Anthropic-Helpful | reward | Helped |
| Use ShiQ instead of CoPG | Cohere R7B · Anthropic-Harmless and Anthropic-Helpful | reward | No clear effect |
| Use ShiQ instead of DPO or DRO | Cohere R7B · Anthropic-Harmless and Anthropic-Helpful | KL divergence | No clear effect |
| Add reward shaping to make reference logits a better initialization (ablation) | ShiQ without initialization shaping (ShiQ-noinit) · LLM experiments | performance | Mixed |
| Add the multi-step extension (ablation) | ShiQ without the multi-step extension (ShiQ-noms) · LLM experiments | performance | Mixed |
| Use token-level rather than sequence-level loss (ablation) | ShiQ sequence-level variant (ShiQ-notk) · LLM experiments | performance | Mixed |
Surprising findings
- ShiQ matches CoPG on HH without knowing which completion is better, using less preference information.
Limitations
- Evaluated only on offline data; no fresh online generations.
- Assumes access to a reliable reward model; no mechanism against reward hacking.
- Evaluation limited to a few LLM benchmarks; not tested on classical RL or robotics.
Experimental setup
Datasets: Anthropic-Harmless Anthropic-Helpful UltraFeedback BFCL-V3 synthetic bandit data synthetic grid-world trajectories
Benchmarks: Anthropic-Harmless Anthropic-Helpful UltraFeedback BFCL-V3
Metrics: reward from reward model KL divergence to reference regret verifiable reward (cumulative per-turn)
Baselines: DPO CoPG DRO multi-turn DPO B-coder
Builds on: path consistency learning direct reward optimization contrastive policy gradient direct preference optimization potential-based reward shaping
Closest papers by meaning
- Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuning (ICLR 2025 · similarity 0.92)
- Efficient Reinforcement Learning with Large Language Model Priors (ICLR 2025 · similarity 0.91)
- Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoning (NeurIPS 2025 · similarity 0.91)
- Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.90)
- Understanding the Effects of RLHF on LLM Generalisation and Diversity (ICLR 2024 · similarity 0.90)
- On Predictability of Reinforcement Learning Dynamics for Large Language Models (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.