Research map / Reinforcement learning / Offline RL
Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LaMo, which initializes a Decision Transformer with pre-trained GPT-2, freezes it with LoRA, swaps linear embeddings for MLPs, and adds an auxiliary language loss to improve offline RL, especially in sparse-reward and low-data settings.
Problem
Offline RL with Decision Transformers is data hungry and underperforms value-based methods like CQL, especially in sparse-reward and limited-data regimes; prior LM-initialized DT (Wiki-RL) did not surpass CQL.
Approach
Use a GPT-2 initialized Decision Transformer with frozen pre-trained attention weights fine-tuned via LoRA, MLP input embeddings and output heads instead of linear projections, plus an auxiliary WikiText language-prediction loss to preserve language ability and prevent overfitting.
Key results
- On sparse-reward Kitchen (1% data, Partial) LaMo averages 31.1 vs 17.9 for Wiki-RL and 10.6 for CQL across low-data settings, an 86% improvement over DT in the low-data table.
- On Atari dense-reward tasks at full data LaMo averages 226.0 vs 189.1 for CQL and 182.6 for DT.
- On MuJoCo at full data LaMo averages 63.3 vs 64.1 for IQL and 63.6 for TD3+BC, i.e. it does not beat the best value-based baselines on dense-reward MuJoCo.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LaMo instead of DT | offline RL on sparse-reward tasks · Sparse-reward tasks, 100% data average | normalized score: 36.5 -> 47.9 | Helped |
| Use LaMo instead of Wiki-RL | offline RL on sparse-reward tasks · Sparse-reward tasks, 1%-50% data average | normalized score: 17.9 -> 31.1 | Helped |
| Use LaMo instead of CQL | offline RL on Atari · Atari, 100% data average | normalized score: 189.1 -> 226.0 | Helped |
| Use LaMo instead of CQL | offline RL on MuJoCo · MuJoCo, 100% data average | normalized score: 63.3 vs 63.1 | Mixed |
| Replace linear projections with MLP embeddings (ablation) | LaMo · offline RL tasks | normalized score | Helped |
| Use LoRA rather than full-weight fine-tuning (ablation) | LaMo · offline RL tasks | normalized score | Helped |
| Add LoRA adaptation rather than freeze all Transformer parameters (ablation) | LaMo · offline RL tasks | normalized score | Helped |
| Add an auxiliary language prediction loss (ablation) | LaMo · Kitchen sparse-reward tasks | normalized score | Helped |
| Add an auxiliary language prediction loss (ablation) | LaMo · tasks outside Kitchen | normalized score | No clear effect |
| Replace GPT-2 language pre-training with ImageGPT visual pre-training (ablation) | LaMo · Atari, 10%-100% data average | normalized score: 165.6 -> 113.6 | Hurt |
| Remove language pre-training and use randomly initialized Transformer weights (ablation) | LaMo with LoRA · Kitchen Partial, 1% data | normalized score: 11.6 -> 9.8 | Hurt |
| Remove language pre-training and use randomly initialized Transformer weights (ablation) | LaMo with LoRA · Kitchen Partial, 10% data | normalized score: 35.1 -> 24.8 | Hurt |
Surprising findings
- A randomly initialized GPT-2 with frozen weights and LoRA still outperforms DT significantly in both sparse- and dense-reward tasks, per the appendix ablation.
- Replacing GPT-2 with ImageGPT pre-training yields much lower results than LaMo on Atari, suggesting sequential language pre-training matters more than visual pre-training.
Limitations
- On dense-reward MuJoCo tasks at larger data scales, LaMo is only comparable to DT and falls behind CQL in Halfcheetah and Walker2d.
- The auxiliary language loss helps mainly in very low-horizon tasks like Kitchen and does not significantly increase performance elsewhere.
- Larger language models were not explored due to computational limits.
Experimental setup
Datasets: D4RL Kitchen D4RL MuJoCo d4rl-atari WikiText Reacher2d (self-generated with PPO)
Benchmarks: D4RL Kitchen D4RL MuJoCo d4rl-atari Reacher2d
Metrics: normalized score
Baselines: Decision Transformer Wiki-RL CQL IQL TD3+BC BCQ NFQ Behavior Cloning Diffusion-QL ImageGPT
Builds on: Decision Transformer LoRA Wiki-RL GPT-2 CQL
Closest papers by meaning
- Unveiling Markov heads in Pretrained Language Models for Offline Reinforcement Learning (ICML 2025 · similarity 0.90)
- Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models (ICLR 2024 · similarity 0.90)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Offline RL by Reward-Weighted Fine-Tuning for Conversation Optimization (NeurIPS 2025 · similarity 0.89)
- Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuning (ICLR 2025 · similarity 0.89)
- On the Modeling Capabilities of Large Language Models for Sequential Decision Making (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.