Research map / Reinforcement learning / RL from feedback
Offline RL by Reward-Weighted Fine-Tuning for Conversation Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReFIT and SWIFT, offline RL methods for LLMs that reduce reward-weighted policy gradients to weighted SFT (with optional reward standardization), and applies them to multi-turn QA conversation policies.
Problem
Learning LLM conversation policies (reasoning or asking clarifying questions in multi-turn QA) from logged trajectories and rewards, without propensity-ratio instability of PPO/GRPO and without reducing rewards to SFT or DPO problems.
Approach
Lower-bound the online RL objective by a reward-weighted log-likelihood of logged trajectories (u >= 1 + log u), which equals weighted SFT over actions with no token-level propensity ratios; SWIFT additionally standardizes rewards using m=3 samples per context.
Key results
- SWIFT achieves 0.7993 accuracy on ARC thinking mode vs 0.6990 for STaR-GATE and 0.3772 for the base model.
- SWIFT wins accuracy in 7 of 12 reasoning experiments and is among the top two in 10 of 12.
- On SciQA thinking mode, REFIT reaches 0.9159 accuracy and the best overall reward of 7.64, versus 0.8186 for STaR-GATE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SWIFT instead of REFIT | Offline QA policy · ARC thinking | accuracy: 0.7889 $\pm$ 0.0240 -> 0.7993 $\pm$ 0.0236 | Helped |
| Use SWIFT instead of REFIT | Offline QA policy · MMLU standard | accuracy: 0.6917 $\pm$ 0.0400 -> 0.7218 $\pm$ 0.0389 | Helped |
| Use SWIFT instead of REFIT | Offline QA policy · MathDial thinking | accuracy: 0.0867 $\pm$ 0.0162 -> 0.1933 $\pm$ 0.0228 | Helped |
| Use SWIFT instead of REFIT | Offline QA policy · CoSQL standard | accuracy: 0.6250 $\pm$ 0.0442 -> 0.6583 $\pm$ 0.0433 | Helped |
| Use SWIFT instead of REFIT | Offline QA policy · MMLU thinking | accuracy: 0.7097 $\pm$ 0.0365 -> 0.7032 $\pm$ 0.0367 | Hurt |
| Use SWIFT instead of REFIT | Offline QA policy · OpenBookQA thinking | accuracy: 0.6504 $\pm$ 0.0317 -> 0.6814 $\pm$ 0.0310 | Helped |
| Increase logged dataset size (ablation) | REFIT · ARC thinking | accuracy: 0.7301 $\pm$ 0.0261 -> 0.7543 $\pm$ 0.0253 | Helped |
| Increase logged dataset size (ablation) | REFIT · ARC standard | accuracy: 0.6715 $\pm$ 0.0326 -> 0.7633 $\pm$ 0.0295 | Helped |
| Increase conversation length from n = 4 to n = 10 (ablation) | SWIFT · OpenBookQA thinking | accuracy: 0.9333 $\pm$ 0.0644 -> 0.6000 $\pm$ 0.1265 | Hurt |
| Increase conversation length from n = 4 to n = 10 (ablation) | REFIT · OpenBookQA thinking | accuracy: 0.9333 $\pm$ 0.0644 -> 0.8667 $\pm$ 0.0878 | Hurt |
| Use REFIT instead of STaR-GATE | Offline QA policy · ARC standard | accuracy: 0.7971 $\pm$ 0.0280 -> 0.7729 $\pm$ 0.0291 | Hurt |
| Use SWIFT instead of STaR-GATE | Offline QA policy · SciQA standard | accuracy: 0.9005 $\pm$ 0.0211 -> 0.9502 $\pm$ 0.0153 | Helped |
Surprising findings
- REFIT underperforms SWIFT substantially on some settings (e.g., thinking OpenBookQA, standard MMLU, standard CoSQL), yet the gap is smaller than expected, attributed to adaptive optimizers in SFT.
- Base model outperforms several offline baselines on reward metrics for some datasets (e.g., MMLU thinking mode base has highest R Overall of 5.87).
Limitations
- RL computational cost is higher than supervised learning, partly mitigated by reduction to SFT.
- Quality of the logged dataset is critical and not studied; diversity obtained only via sampling temperatures.
- No human evaluation; judgments rely on GPT-4o and Claude 4 Opus LLM judges.
Experimental setup
Datasets: OpenBookQA ARC SciQA MMLU CoSQL MathDial
Benchmarks: OpenBookQA ARC SciQA MMLU CoSQL MathDial
Metrics: accuracy thinking rate GPT-4o judge reward overall reasoning comprehensiveness pedagogic value confidence calibration
Baselines: base policy STaR-GATE STaR-GATE distilled Step-DPO DPO
Builds on: STaR-GATE DPO GRPO PPO reward-weighted regression
Closest papers by meaning
- Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuning (ICLR 2025 · similarity 0.90)
- Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards (NeurIPS 2025 · similarity 0.89)
- Planning without Search: Refining Frontier LLMs with Offline Goal-Conditioned RL (NeurIPS 2025 · similarity 0.89)
- Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learning (ICLR 2024 · similarity 0.89)
- Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models (ICLR 2024 · similarity 0.89)
- The Importance of Online Data: Understanding Preference Fine-tuning via Coverage (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.