Research map / Reinforcement learning / Offline RL
Behavioral Entropy-Guided Dataset Generation for Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends behavioral entropy (BE) to continuous spaces with k-NN estimators and RL reward, then uses BE-maximizing APT policies to generate MuJoCo datasets that yield better offline RL than Shannon, Rényi, RND and SMM datasets.
Problem
Dataset generation for offline RL needs exploration objectives that yield diverse state-space coverage; Shannon and Rényi entropies give limited or unstable coverage, and behavioral entropy had only been defined for discrete distributions.
Approach
Define differential behavioral entropy with Prelec probability weighting for continuous densities, derive k-NN estimators with convergence and finite-sample bias/variance bounds, and convert it into a per-step reward (with d=1 and a constant c) so standard RL (APT) maximizes BE of state occupancy for generating offline RL datasets.
Key results
- Best offline RL performance on BE datasets exceeds SE, RND and SMM datasets on all five tasks, e.g. Walker Walk 904.66 (BE) vs 895.89 (SE) and 735.77 (RND).
- BE datasets beat Rényi datasets on four of five tasks; Rényi wins on Walker Run (440.53 vs 385.07).
- BE-generated offline RL performance is stabler across the alpha parameter than Rényi across q, which is unstable and gives poor coverage for q > 1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Generate offline datasets using BE rather than Shannon entropy | Offline RL dataset generation · Walker Stand, Walker Walk, Walker Run, Quadruped Walk, Quadruped Run | offline RL performance | Helped |
| Generate offline datasets using BE rather than RND | Offline RL dataset generation · Walker and Quadruped tasks | offline RL performance | Helped |
| Generate offline datasets using BE rather than SMM | Offline RL dataset generation · Walker and Quadruped tasks | offline RL performance | Helped |
| Generate offline datasets using BE rather than Rényi entropy | Offline RL dataset generation · Walker and Quadruped tasks | offline RL performance | Mixed |
| Use the best-performing Rényi parameter rather than BE | Offline RL dataset generation · Walker Run | offline RL performance | Mixed |
| Use RND-generated datasets rather than BE-generated datasets | Offline RL dataset generation · Quadruped tasks | offline RL performance | No clear effect |
| Increase the Shannon-generated dataset size from 500K to 3M elements | Shannon entropy dataset generation · Walker Stand and Walker Run | offline RL performance | Helped |
| Increase the Shannon-generated dataset size from 500K to 3M elements | Shannon entropy dataset generation · Quadruped tasks and TD3 on Walker Walk | offline RL performance | Hurt |
| Increase offline RL training from 100K to 200K steps (ablation) | Offline RL on a 3M-element Shannon entropy dataset · Downstream tasks | offline RL performance | No clear effect |
Surprising findings
- Rényi entropy with q=1.1 is outperformed by BE on most tasks, and Rényi coverage is poor for q>1.
- A 3M-element Shannon dataset does not uniformly beat the 500K Shannon dataset; it is worse on both Quadruped tasks and TD3 Walker Walk.
Limitations
- Experiments limited to Walker and Quadruped MuJoCo environments and three offline RL algorithms due to computational cost.
- Estimator bias is not corrected for finite k; theory gives only probabilistic bounds, and k is capped at 15 with dimension reduction in practice.
- Rényi datasets for q in {2,3,5} omitted from offline RL experiments.
Experimental setup
Datasets: URLB Walker URLB Quadruped ExORL datasets
Benchmarks: URLB Walker Stand URLB Walker Walk URLB Walker Run URLB Quadruped Walk URLB Quadruped Run
Metrics: best offline RL episode return t-SNE and PHATE coverage visualizations
Baselines: Shannon entropy Rényi entropy RND SMM
Builds on: Active Pre-Training (APT) ExORL URLB behavioral entropy (discrete) Rényi entropy exploration
Closest papers by meaning
- Mind Your Entropy: From Maximum Entropy to Trajectory Entropy-Constrained RL (ICML 2026 · similarity 0.90)
- SEMDICE: Off-policy State Entropy Maximization via Stationary Distribution Correction Estimation (ICLR 2025 · similarity 0.89)
- Simple Ingredients for Offline Reinforcement Learning (ICML 2024 · similarity 0.89)
- Efficient Policy Evaluation with Offline Data Informed Behavior Policy Design (ICML 2024 · similarity 0.89)
- OCEAN-MBRL: Offline Conservative Exploration for Model-Based Offline Reinforcement Learning (AAAI 2024 · similarity 0.89)
- Hybrid Reinforcement Learning from Offline Observation Alone (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.