Research map / Reinforcement learning / Offline RL
Offline Behavior Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formulates offline behavior distillation (OBD) to synthesize a few expert (state, action) pairs from sub-optimal D4RL data, and proposes action-value weighted PBC (Av-PBC), which gives a linear-discount-complexity performance guarantee and outperforms DBC and PBC.
Problem
Massive sub-optimal offline RL datasets make policy training inefficient; the goal is to distill them into a small set of expert-like state-action pairs that enable rapid behavioral cloning, but no reliable offline objective exists to measure distillation quality.
Approach
Replace the TV-distance policy-based BC objective (quadratic discount complexity) with an action-value weighted decision difference, whose gap to policy performance is exactly J(pi*)-J(pi) = 1/(1-gamma) E_{s~d_pi}[sum_a q_{pi*}(s,a)(pi*-pi)], yielding linear discount complexity; practically sample a~pi* and weight squared differences by q_{pi*}.
Key results
- Av-PBC averages 38.2 normalized return across nine D4RL datasets vs 30.4 for PBC and 20.9 for DBC, a 25.7% gain over PBC and 82.8% over DBC.
- Av-PBC needs about a quarter of the distillation steps of DBC and PBC, and BC on 256 distilled pairs trains in 100 steps (0.2 s) versus 25k-125k steps on the full data.
- Ensembling 10 policies trained on Av-PBC-distilled data raises average return to 48.1, a 9.9-point gain.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use action-value-weighted PBC instead of PBC | Offline behavior distillation · D4RL (9 datasets), average | normalized return: 30.4 -> 38.2 | Helped |
| Correct offline data actions with the near-optimal policy (PBC instead of DBC) | Offline behavior distillation · D4RL (9 datasets), average | normalized return: 20.9 -> 30.4 | Helped |
| Use Av-PBC instead of DBC | Offline behavior distillation · D4RL (9 datasets), average | normalized return: 20.9 -> 38.2 | Helped |
| Train an ensemble of 10 policies on Av-PBC-distilled data | Av-PBC · D4RL (9 datasets), average | normalized return: +9.9 | Helped |
| Increase synthetic data size from 16 to 256 (ablation) | Av-PBC · D4RL, Halfcheetah M-R | normalized return: 6.9 -> 35.9 | Helped |
| Increase synthetic data size from 16 to 256 (ablation) | Av-PBC · D4RL, Hopper M-R | normalized return: 27.3 -> 40.9 | Helped |
| Increase synthetic data size from 16 to 256 (ablation) | Av-PBC · D4RL, Walker2D M-R | normalized return: 14.8 -> 55.0 | Helped |
| Use a 3-layer policy network instead of the default 4-layer MLP (ablation) | Av-PBC-distilled data · D4RL (9 datasets), average | normalized return: 38.2 -> 41.0 | Mixed |
| Use a 6-layer policy network instead of the default 4-layer MLP (ablation) | Av-PBC-distilled data · D4RL (9 datasets), average | normalized return: 38.2 -> 31.7 | Hurt |
| Use Adam instead of the default SGD optimizer (ablation) | Av-PBC-distilled data · D4RL (9 datasets), average | normalized return: 38.2 -> 39.3 | Helped |
| Use AdamW instead of the default SGD optimizer (ablation) | Av-PBC-distilled data · D4RL (9 datasets), average | normalized return: 38.2 -> 39.3 | Helped |
| Use SGDm instead of the default SGD optimizer (ablation) | Av-PBC-distilled data · D4RL (9 datasets), average | normalized return: 38.2 -> 38.1 | No clear effect |
Surprising findings
- Av-PBC distilled from medium-replay data outperforms data distilled from medium-expert data, which the authors attribute to greater state coverage in medium-replay.
- Av-PBC on medium-replay surpasses BC trained on the whole dataset on average-level comparisons (e.g., Halfcheetah M-R 35.9 vs BC whole 14.0).
Limitations
- Distilled data are (state, action) pairs without rewards, so they only work with behavior cloning, not Bellman-based RL algorithms.
- A notable performance gap remains between OBD (38.2) and offline RL on the whole dataset (70.1).
- Distillation is computationally expensive due to bi-level optimization (about 25 hours on a V100 for 50k steps).
Experimental setup
Datasets: D4RL halfcheetah-medium-replay D4RL halfcheetah-medium D4RL halfcheetah-medium-expert D4RL hopper-medium-replay D4RL hopper-medium D4RL hopper-medium-expert D4RL walker2d-medium-replay D4RL walker2d-medium D4RL walker2d-medium-expert
Benchmarks: D4RL Halfcheetah D4RL Hopper D4RL Walker2D
Metrics: normalized return
Baselines: Random Selection DBC PBC BC (Whole) Cal-QL (Whole)
Builds on: dataset distillation Cal-QL behavior distillation backpropagation through time
Closest papers by meaning
- Iteratively Refined Behavior Regularization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.91)
- Efficient Policy Evaluation with Offline Data Informed Behavior Policy Design (ICML 2024 · similarity 0.90)
- Is Value Learning Really the Main Bottleneck in Offline RL? (NeurIPS 2024 · similarity 0.90)
- Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning (AAAI 2024 · similarity 0.90)
- Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning (ICML 2024 · similarity 0.90)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.