Research map / Reinforcement learning / Visual and representation RL
Behaviour Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formalizes behaviour distillation (condensing an expert policy into a few synthetic state-action pairs without expert data) and proposes HaDES, which evolves such datasets with evolution strategies over a behaviour-cloning inner loop.
Problem
Dataset distillation methods assume an expert or fixed dataset, so they cannot be applied to reinforcement learning; the goal is to distill the information needed to train an expert policy into a tiny synthetic state-action dataset without expert data.
Approach
Bi-level optimization where the synthetic dataset is evolved by OpenES/SNES in the outer loop, with the fitness being the return of a policy trained by behaviour cloning on that dataset in the inner loop; a randomized-initialization variant (HaDES-R) yields datasets that generalize across architectures and hyperparameters, and the same approach doubles as a memory-efficient neuroevolution parametr
Key results
- HaDES-F matches or outperforms direct ES on all Brax environments and outperforms ES on Breakout-MinAtar, using datasets of 64 (Brax) or 16 (MinAtar) state-action pairs.
- HaDES-R achieves higher return than ES neuroevolution on 6 of 8 Brax environments and generalizes across policy widths and hyperparameters on Hopper.
- On FashionMNIST with 1 image per class, HaDES reaches 80.2% test accuracy versus 78.6% for RFAD, while on MNIST it trails RFAD (90.1% vs 94.4%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HaDES instead of RFAD for 1 image/class distillation | MNIST classifiers · MNIST, 1 image/class | test accuracy: 94.4 ± 1.5 -> 90.1 ± 0.3 | Hurt |
| Use HaDES instead of RFAD for 1 image/class distillation | FashionMNIST classifiers · FashionMNIST, 1 image/class | test accuracy: 78.6 ± 1.3 -> 80.2 ± 0.4 | Helped |
| Randomize policy initialization during dataset evolution instead of using a fixed initialization (ablation) | HaDES behaviour-distillation datasets · Hopper; transfer across architectures and training parameters | policy return/generalization | Helped |
| Use a fixed policy initialization instead of randomized initializations (ablation) | HaDES neuroevolution · Brax and MinAtar environments | policy return | Helped |
| Use randomized policy initialization instead of a fixed initialization (ablation) | HaDES neuroevolution · Brax and MinAtar environments | policy return versus ES: 6/8 | Mixed |
| Use HaDES-F instead of direct ES neuroevolution | Neuroevolution · Brax environments | policy return | Helped |
| Use HaDES-F instead of direct ES neuroevolution | Neuroevolution · MinAtar environments | policy return | Mixed |
| Use a dataset that is too small to capture the expert behaviour (ablation) | HaDES policies · Humanoid | return: 1013 | Hurt |
| Increase the dataset size to 256 state-action pairs (ablation) | HaDES policies · HaDES distillation-budget experiments | return: 256 | Hurt |
| Train on the merged, zero-padded datasets rather than the single-task dataset | HaDES multi-task agents · Halfcheetah and Hopper | normalized fitness: ~ 50% | Mixed |
| Train on the merged, zero-padded datasets rather than the single-task dataset | HaDES multi-task agents · Humanoid and Humanoid Standup | normalized fitness: ≳ 100% | No clear effect |
Surprising findings
- HaDES-R, which optimizes for generalization, yields lower policy returns in neuroevolution than HaDES-F, as expected from its weaker inductive bias.
- Humanoid-Standup policies exploited a collision physics glitch to achieve extremely high returns.
Limitations
- Computationally expensive since evolutionary methods require large populations.
- Number of synthetic parameters grows linearly with dataset size, so high-dimensional pixel environments and CIFAR-10 scaling are hard.
- Many hyperparameters to tune in both outer ES loop and inner supervised loop; large datasets (256) degrade performance.
Experimental setup
Datasets: Brax environments MinAtar environments MNIST FashionMNIST
Benchmarks: Brax continuous control (8 environments) MinAtar (4 environments) MNIST 1 img/cls FashionMNIST 1 img/cls
Metrics: episodic return normalized fitness test accuracy seconds per generation
Baselines: ES neuroevolution PPO RFAD
Builds on: dataset distillation evolution strategies OpenES SNES PureJaxRL
Closest papers by meaning
- Offline Behavior Distillation (NeurIPS 2024 · similarity 0.88)
- How Ensembles of Distilled Policies Improve Generalisation in Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
- Behavioral Entropy-Guided Dataset Generation for Offline Reinforcement Learning (ICLR 2025 · similarity 0.86)
- Foundation Policies with Hilbert Representations (ICML 2024 · similarity 0.86)
- What is Dataset Distillation Learning? (ICML 2024 · similarity 0.86)
- Privileged Information Distillation for Language Models (ICML 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.