Research map / Reinforcement learning / Offline RL
Pre-training with Synthetic Data Helps Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that pre-training Decision Transformer and CQL on simple synthetic data (Markov chain or MDP sequences, or IID data) before offline RL fine-tuning matches or beats language (Wiki) pre-training on D4RL MuJoCo tasks.
Problem
Whether the performance gains from pre-training offline RL models on a language corpus are specific to language, or can be obtained from simple synthetic pre-training data, for both transformer (Decision Transformer) and MLP (CQL) backbones.
Approach
Pre-train with autoregressive next-state prediction on data from a randomly parameterized one-step Markov chain (100 states) for DT, and with forward-dynamics prediction on synthetic MDP data for CQL's MLP Q-networks; the synthetic data needs no relation to the environment, and even IID data works, which is explained by a centroid-based analysis of the forward-dynamics objective.
Key results
- DT+Synthetic (1-step MC, 20K pre-training updates) reaches 73.6 average final score vs 67.7 for DT and 69.9 for DT+Wiki across 12 datasets.
- CQL+MDP with 1,000 states/actions improves the CQL average from 78.0 to 85.6 (about 10%), with up to 84% gains on a medium-expert dataset.
- Synthetic MC pre-training with only 1K updates already matches DT+Wiki average performance (70.5 vs 69.9), and 100K-update CQL synthetic pre-training takes about 4 minutes on a single GPU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pre-train DT on one-step MC synthetic data rather than training from scratch | Decision Transformer · D4RL MuJoCo, average over datasets | normalized test score (final): 67.7 $\pm$ 5.4 -> 73.6 $\pm$ 4.9 | Helped |
| Pre-train DT on one-step MC synthetic data rather than Wikipedia | Decision Transformer · D4RL MuJoCo, average over datasets | normalized test score (final): 69.9 $\pm$ 5.6 -> 73.6 $\pm$ 4.9 | Helped |
| Pre-train CQL on synthetic MDP data | CQL · D4RL MuJoCo, average over datasets | normalized test score (final): 78.0 $\pm$ 4.7 -> 84.9 $\pm$ 3.7 | Helped |
| Increase MC history from 1 step to 2 or 5 steps (ablation) | DT+Synthetic · D4RL MuJoCo, average over datasets | normalized test score (final): 73.6 $\pm$ 4.9 -> 73.0 $\pm$ 4.2 (2-MC); 71.8 $\pm$ 4.8 (5-MC) | Hurt |
| Use MC state spaces of 10, 10,000, or 100,000 rather than 100 (ablation) | DT+Synthetic · D4RL MuJoCo, average over datasets | normalized test score (final): 73.6 $\pm$ 4.9 -> 71.9 $\pm$ 5.3 (S10); 72.9 $\pm$ 4.6 (S10000); 72.4 $\pm$ 4.7 (S100000) | Mixed |
| Increase MC state space from 100 to 1,000 (ablation) | DT+Synthetic · D4RL MuJoCo, average over datasets | normalized test score (final): 73.6 $\pm$ 4.9 -> 74.0 $\pm$ 4.0 | Helped |
| Vary MC transition temperature from 1 to 0.01, 0.1, 10, or 100 (ablation) | DT+Synthetic · D4RL MuJoCo, average over datasets | normalized test score (final): 73.6 $\pm$ 4.9 -> 71.7 $\pm$ 4.7 (τ=0.01); 73.1 $\pm$ 5.4 (τ=0.1); 72.4 $\pm$ 4.7 (τ=10); 71.9 $\pm$ 4.7 (τ=100) | Hurt |
| Use IID uniform synthetic states instead of MC-generated states (ablation) | DT+Synthetic · D4RL MuJoCo, average over datasets | normalized test score (final): 73.6 $\pm$ 4.9 -> 71.3 $\pm$ 5.1 | Hurt |
| Reduce DT synthetic pre-training from 20k to 1k or 10k updates (ablation) | DT+Synthetic · D4RL MuJoCo, average over datasets | normalized test score (final): 73.6 $\pm$ 4.9 -> 70.5 $\pm$ 5.5 (1k); 71.2 $\pm$ 5.1 (10k) | Hurt |
| Increase DT synthetic pre-training from 20k to 40k or 60k updates (ablation) | DT+Synthetic · D4RL MuJoCo, average over datasets | normalized test score (final): 73.6 $\pm$ 4.9 -> 74.4 $\pm$ 5.2 (40k); 74.2 $\pm$ 5.1 (60k) | Helped |
| Increase DT synthetic pre-training from 20k to 80k updates (ablation) | DT+Synthetic · D4RL MuJoCo, average over datasets | normalized test score (final): 73.6 $\pm$ 4.9 -> 73.8 $\pm$ 4.6 | Mixed |
| Use CQL synthetic MDP state/action space sizes other than 100 (ablation) | CQL+MDP · D4RL MuJoCo, average over datasets | normalized test score (final): 84.9 $\pm$ 3.7 -> 83.7 $\pm$ 3.8 (S=10); 85.6 $\pm$ 3.0 (S=1,000); 85.2 $\pm$ 3.5 (S=10,000); 84.6 $\pm$ 4.0 (S=100,000) | Mixed |
Surprising findings
- IID uniform synthetic data pre-training performs about as well as Markov-chain or MDP synthetic data for both DT and CQL.
- A one-step Markov chain with 100 states outperforms pre-training on the Wikipedia corpus for DT, despite having a far smaller vocabulary and no long-range structure.
Limitations
- Evaluated only on D4RL MuJoCo locomotion tasks (halfcheetah, hopper, walker2d, ant).
- Authors do not explain fully why unrelated pre-training helps; they suspect an optimization smoothing effect.
- The theoretical analysis is partial, assuming a very expressive MLP and a centroid argument.
Experimental setup
Datasets: D4RL halfcheetah, hopper, walker2d, ant (medium-expert, medium-replay, medium) WikiText-103 (Wiki pre-training baseline) synthetic Markov chain data synthetic MDP data
Benchmarks: D4RL halfcheetah D4RL hopper D4RL walker2d D4RL ant
Metrics: normalized score (D4RL) average final performance over last four evaluations best score number of updates to reach 90% of baseline wall-clock computation time
Baselines: Decision Transformer DT+Wiki CQL CQL+IID
Builds on: Decision Transformer Conservative Q-Learning Wikipedia pre-training for DT (Reid et al., 2022) Safe Q Target
Closest papers by meaning
- Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data (ICLR 2025 · similarity 0.89)
- When should we prefer Decision Transformers for Offline Reinforcement Learning? (ICLR 2024 · similarity 0.88)
- Online Pre-Training for Offline-to-Online Reinforcement Learning (ICML 2025 · similarity 0.88)
- Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learning (ICLR 2024 · similarity 0.88)
- Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuning (ICLR 2025 · similarity 0.88)
- A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.