Research map / Reinforcement learning / Offline RL
Learning from Sparse Offline Datasets via Conservative Density Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Conservative Density Estimation (CDE), which applies pessimism directly to the stationary state-action distribution via f-divergence-regularized marginal importance sampling with an OOD mixture proposal, improving offline RL under sparse rewards and scarce data.
Problem
Offline RL suffers from OOD extrapolation errors, especially with sparse rewards or scarce data; DICE-style marginal importance sampling becomes unstable due to support mismatch between the dataset and the learned policy's stationary distribution.
Approach
Impose conservatism on the stationary state-action distribution by constraining its density on OOD state-actions (d^pi <= eps*mu), and use a mixture proposal of dataset and OOD distributions in importance sampling so the importance ratio is bounded; the value and policy are then extracted in closed form from the optimal ratio.
Key results
- On D4RL sparse-reward Maze2D, CDE reaches 134.1 / 146.1 / 210.0 normalized score on umaze / medium / large, versus 111.0 / 145.2 / 155.7 for OptiDICE.
- On sparse-MuJoCo medium-expert tasks, CDE gets 97.0 on walker2d-medium-expert vs 69.0 for OptiDICE and 94.6 for IQL.
- CDE keeps high performance with only 1% of trajectories where baselines degrade, with OptiDICE showing sharp drops in scarce data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CDE instead of OptiDICE | Offline RL · maze2d-umaze | normalized score: 111.0 -> 134.1 | Helped |
| use CDE instead of OptiDICE | Offline RL · maze2d-medium | normalized score: 145.2 -> 146.1 | Helped |
| use CDE instead of OptiDICE | Offline RL · maze2d-large | normalized score: 155.7 -> 210.0 | Helped |
| use CDE instead of IQL | Offline RL · pen-human | normalized score: 71.5 -> 72.1 | Helped |
| use CDE instead of CQL | Offline RL · hammer-human | normalized score: 4.4 -> 1.9 | Hurt |
| use CDE instead of CQL | Offline RL · door-human | normalized score: 9.9 -> 7.7 | Hurt |
| use CDE instead of CQL | Offline RL · relocate-human | normalized score: 0.2 -> 0.3 | Helped |
| use CDE instead of BCQ | Offline RL · pen-expert | normalized score: 114.9 -> 105.0 | Hurt |
| use CDE instead of OptiDICE | Offline RL · hammer-expert | normalized score: 127.0 -> 126.3 | Hurt |
| use CDE instead of IQL | Offline RL · door-expert | normalized score: 103.8 -> 105.9 | Helped |
| use CDE instead of IQL | Offline RL · relocate-expert | normalized score: 102.7 -> 102.6 | Hurt |
| use CDE instead of CQL | Offline RL · halfcheetah-medium | success rate (%): 97.6 -> 82.0 | Hurt |
Surprising findings
- OptiDICE, competitive with full data, degrades sharply in scarce-data settings due to support mismatch in importance sampling.
Limitations
- Requires strict alignment of initial state distribution between offline data and online environment.
- Performance sensitive to the conservatism level; overly strict constraints trap the agent.
Experimental setup
Datasets: D4RL Maze2D D4RL Adroit D4RL MuJoCo
Benchmarks: D4RL Maze2D D4RL Adroit sparse-MuJoCo
Metrics: normalized score success rate
Baselines: BC BCQ CQL IQL TD3+BC AlgaeDICE OptiDICE
Builds on: DICE OptiDICE Conservative Q-learning SAC D4RL
Closest papers by meaning
- Dynamic Uncertainty Estimation for Offline Reinforcement Learning (AAAI 2025 · similarity 0.90)
- Conservative State Value Estimation for Offline Reinforcement Learning (NeurIPS 2023 · similarity 0.90)
- Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian (ICLR 2023 · similarity 0.90)
- Fewer May Be Better: Enhancing Offline Reinforcement Learning with Reduced Dataset (ICLR 2025 · similarity 0.88)
- Harnessing Density Ratios for Online Reinforcement Learning (ICLR 2024 · similarity 0.88)
- Relaxed Stationary Distribution Correction Estimation for Improved Offline Policy Optimization (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.