Research map / Reinforcement learning / Offline RL
Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Q-SFT, an offline RL method that casts Q-learning as a weighted cross-entropy fine-tuning of LLM/VLM token probabilities so that pretrained logits directly encode conservative Q-values, without new value heads.
Problem
Value-based offline RL for multi-turn LLM/VLM fine-tuning scales poorly and discards pretrained likelihoods because it requires regressing Q-values via new heads and TD losses.
Approach
Replace the TD regression loss with a weighted cross-entropy where weights are Bellman targets computed from the behavior-normalized probabilities of the model's own outputs, so token probabilities represent conservative Q-values initialized from pretrained logits; policy is extracted at inference as pi_phi(a|s) * exp(beta * p_theta(a|s)).
Key results
- Q-SFT achieves best or near-best scores on Chess (0.15), Wordle (-2.11), and 20Q (-13.1) versus SFT, ILQL, and ReAct.
- On ALFWorld, Q-SFT attains 21 Examine, 19 Clean, 18 Cool success rates, outperforming SFT and ILQL on most task types.
- With GPT2-xl on 20Q using 10% data, Q-SFT significantly outperforms ILQL as model size grows.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Q-SFT instead of SFT | Chess language game · Chess | average score: 0.11 -> 0.15 | Helped |
| Use Q-SFT instead of ILQL | Wordle language game · Wordle | average score: -2.08 -> -2.11 | Hurt |
| Use Q-SFT instead of ReAct | 20Q language game · 20Q | average score: -13.2 -> -13.1 | Helped |
| Use Q-SFT instead of ReAct | ALFWorld policy · ALFWorld tasks | success rate | Mixed |
| Use Q-SFT instead of SFT | ALFWorld policy · ALFWorld tasks | success rate | Mixed |
| Use Q-SFT instead of ILQL | ALFWorld policy · ALFWorld tasks | success rate | Helped |
Surprising findings
- Value-based baseline ILQL underperforms SFT on several ALFWorld task types, suggesting regression-based Q-learning scales poorly with pretrained VLMs.
Limitations
- Conservativeness guarantee only holds for actions with non-negligible Q-values (Q* >= 1/(|A|-1))
- Assumes bounded total rewards (sum of discounted rewards <= 1)
- Only evaluated offline; online adaptation left to future work
Experimental setup
Datasets: LMRL benchmark (Chess, Wordle, 20 Questions) WebShop ALFWorld large-scale robotic manipulation dataset
Benchmarks: Chess (LMRL) Wordle (LMRL) 20 Questions (LMRL) ALFWorld robotic manipulation
Metrics: average return success rate
Baselines: ReAct SFT ILQL Q-Transformer CQL ArCHer
Builds on: ILQL Conservative Q-Learning Decision Transformer ArCHer
Closest papers by meaning
- ShiQ: Bringing back Bellman to LLMs (NeurIPS 2025 · similarity 0.92)
- Offline RL by Reward-Weighted Fine-Tuning for Conversation Optimization (NeurIPS 2025 · similarity 0.90)
- ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL (ICML 2024 · similarity 0.90)
- Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Instead (ICLR 2026 · similarity 0.89)
- Vanishing Gradients in Reinforcement Finetuning of Language Models (ICLR 2024 · similarity 0.89)
- LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.