Research map / Reinforcement learning / RL from feedback
Imitating Language via Scalable Inverse Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reformulates inverse soft-Q-learning (IQLearn) as a temporal-difference-regularized MLE objective and evaluates offline/online IRL against MLE and GAIL for fine-tuning T5 and PaLM2 models, showing better quality-diversity trade-offs.
Problem
Maximum likelihood (next-token prediction) dominates LM imitation but suffers from compounding errors, exposure bias and limited generation diversity; it is unclear when inverse RL can be an effective alternative for supervised fine-tuning.
Approach
Rewrite inverse soft-Q-learning in terms of state values and policy logits so it becomes MLE plus a dynamics-dependent temporal-difference regularizer whose weight lambda can be annealed (lambda=0 recovers MLE), enabling cheap offline training on fixed SFT data.
Key results
- Offline IQLearn improves BLEU over behavior cloning on WMT22 by up to +2.16 with PaLM2 (dev/test tables), and gives consistent gains in ROUGE/accuracy on T5 and PaLM2 across tasks.
- IQLearn increases generation diversity (lower Self-BLEU / higher entropy) relative to MLE at comparable task performance, forming a better quality-diversity Pareto front.
- Online IQLearn (alpha=0.1) extracted rewards correlate with task metrics (Spearman 0.64 ROUGE-1 on TLDR vs -0.05 for MLE).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use offline IQLearn instead of MLE | PaLM2 Gecko · WMT22 test | BLEU: 32.34 -> 35.68 | Helped |
| Use IRL-based imitation instead of MLE | T5 models · GSM8k and XSUM | task performance | Helped |
| Use IQLearn instead of MLE | PaLM2 Gecko · GSM8k, TLDR, and WMT22 | task performance | Helped |
| Add temporal-difference regularization with IQLearn (ablation) | T5-base trained on smaller XSUM subsets · XSUM | overfitting robustness | Helped |
| Add online samples to IQLearn (ablation) | T5-base · GSM8k | generation diversity | Helped |
| Add MLE loss to GAIL (ablation) | T5-Large · XSUM | ROUGE-1 | Helped |
| Increase the IQLearn regularization coefficient to very high values (ablation) | PaLM2 Gecko · WMT22 | BLEU | Hurt |
| Use online IQLearn rewards instead of MLE rewards | IQLearn reward model · TLDR ROUGE-1 | Spearman correlation: -0.05 -> 0.64 | Helped |
| Use online IQLearn rewards instead of MLE rewards | IQLearn reward model · WMT22 BLEU | Spearman correlation: -0.05 -> 0.43 | Helped |
| Use online IQLearn rewards instead of MLE rewards | IQLearn reward model · GSM8k accuracy with calculator | Spearman correlation: -0.02 -> 0.17 | Helped |
| Increase TD regularization λ (ablation) | Online IQLearn reward model · WMT22 | reward-metric alignment | Helped |
| Train online rather than offline (ablation) | IQLearn in the toy MDP without recovery · Toy sequence-generation MDP | success rate: over 11% | No clear effect |
Surprising findings
- Offline IQLearn achieves most of the gains without online sampling; online data adds only limited performance benefit, attributed to autoregressive concatenation dynamics preventing exact recovery from mistakes.
- GAIL was hard to stabilize for PaLM2 and required starting from an MLE checkpoint for T5; without an MLE loss GAIL agents often generate maximum-length sequences.
Limitations
- Diversity measured only by Self-BLEU and entropy, not by downstream RLHF benefit
- Online IRL gains limited in single-turn autoregressive setting; GAIL hard to stabilize, especially for PaLM2
- Only one lambda value per task for PaLM2 experiments due to compute cost
Experimental setup
Datasets: XSUM GSM8k TLDR WMT22
Benchmarks: XSUM GSM8k TLDR WMT22
Metrics: ROUGE-1 ROUGE-2 ROUGE-LSUM accuracy BLEU ChrF Self-BLEU per-token entropy Spearman rank correlation
Baselines: MLE GAIL IQLearn (online) entropy-regularized MLE
Builds on: IQLearn GAIL behavior cloning soft Q-learning
Closest papers by meaning
- Approximated Variational Bayesian Inverse Reinforcement Learning for Large Language Model Alignment (AAAI 2025 · similarity 0.91)
- On a Connection Between Imitation Learning and RLHF (ICLR 2025 · similarity 0.90)
- Hybrid Inverse Reinforcement Learning (ICML 2024 · similarity 0.90)
- Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuning (ICLR 2025 · similarity 0.89)
- Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignment (NeurIPS 2024 · similarity 0.89)
- Dense Reward for Free in Reinforcement Learning from Human Feedback (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.