Research map / Reinforcement learning / Offline RL
CUDC: A Curiosity-Driven Unsupervised Data Collection Method with Adaptive Temporal Distances for Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CUDC, a curiosity-driven unsupervised data collection method that adapts the temporal distance k via a self-supervised reachability module, yielding task-agnostic datasets for multi-task offline RL on DeepMind control tasks.
Problem
Collecting task-agnostic datasets for multi-task offline RL is hard; existing unsupervised data collectors (ExORL) fix the temporal distance k=3 between current and future states, limiting feature representation diversity.
Approach
Estimate the probability that a k-step future state is reachable from the current state with a contrastive self-supervised reachability module, increase k adaptively when most tuples have low curiosity weight, and use a mixed intrinsic reward (KNN state-action entropy plus prediction error) and curiosity-weighted DDPG updates.
Key results
- CUDC achieves higher 500K-step offline TD3 scores than all baselines in all 12 downstream tasks (e.g., Walker Walk 893 vs 845 for APS best baseline).
- Outperforms APT by 51% in the Quadruped domain and APS by 6% in Walker, and ICM by 10% in Jaco Arm (domain averages).
- Adapting k alone (CUDC_vary^ICM) yields 1.25x computational efficiency at 100K and 1.16x learning performance at 500K over ICM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CUDC-collected data instead of the strongest baseline data | TD3 offline learning · 12 DeepMind Control downstream tasks | score at 500K steps: CUDC ... achieve better learning performance (500K) in all 12 downstream tasks | Helped |
| adapt temporal distance using ICM intrinsic rewards (ablation) | ICM data collection · 4 Walker tasks | computational efficiency at 100K steps: 1.25 × | Helped |
| adapt temporal distance using ICM intrinsic rewards (ablation) | ICM data collection · 4 Walker tasks | offline learning performance at 500K steps: 1.16 × | Helped |
| adapt temporal distance using APT intrinsic rewards (ablation) | APT data collection · 4 Walker tasks | score at 100K and 500K steps: 1.12 × and 1.04 × | Helped |
| add mixed state-action entropy and prediction-error intrinsic reward (ablation) | CUDC_vary_APT · 4 Walker tasks | offline learning score: 3.3%; 3.0% | Helped |
| add mixed state-action entropy and prediction-error intrinsic reward (ablation) | CUDC_vary_APT · 4 Walker tasks | standard deviation at 100K steps: 67% increase | Hurt |
| add full reachability module without critic-actor regularization (ablation) | CUDC_reward · 4 Walker tasks | standard deviation at 100K and 500K steps: 48% and 25% | Helped |
| add curiosity-weight regularization to critic-actor updates (ablation) | CUDC_reach · 4 Walker tasks | learning efficiency and capability: 3.2% and 4.0% | Helped |
| sample k uniformly from 3 to 6 instead of curiosity-driven adaptation (ablation) | CUDC · 4 Walker tasks | overall learning capability at 500K steps: 85%; 300% increase in standard deviation | Hurt |
| sample k normally with increasing mean instead of curiosity-driven adaptation (ablation) | CUDC · 4 Walker tasks | overall performance at 500K steps: 4.5% weaker; 128% higher | Hurt |
| use adaptive temporal distance instead of a fixed k=3 (ablation) | CUDC · Walker tasks | offline learning performance | Helped |
Surprising findings
- Computational efficiency (100K steps) is poor in the Jaco Arm domain, where CUDC does not clearly beat baselines.
- Uniform or normal sampling of k performs worse than curiosity-driven adaptation, but the standard deviation increases for varying-k variants.
Limitations
- Scalability to complex environments may be limited, especially in Jaco Arm where computational efficiency is poor.
- k is manually restricted to range 3-6 with bounds on regularization weights for stability; the schedule is heuristic.
- Std deviation increases in some varying-k variants, indicating instability.
Experimental setup
Datasets: ExORL datasets collected on DeepMind control suite (Walker, Quadruped, Jaco Arm)
Benchmarks: DeepMind control suite 12 downstream tasks (Walker, Quadruped, Jaco Arm)
Metrics: episode return score at 100K steps (efficiency) score at 500K steps (learning performance) standard deviation across 5 seeds
Baselines: Random APS ProtoRL ICM APT Explore2Offline
Builds on: ExORL DDPG APT ICM ProtoRL
Closest papers by meaning
- Fewer May Be Better: Enhancing Offline Reinforcement Learning with Reduced Dataset (ICLR 2025 · similarity 0.87)
- DaCe-DT: Data-Centric Offline Multi-Task Reinforcement Learning via Adaptive Prompts and Trajectory Correction for Heterogeneous Tasks (NeurIPS 2026 · similarity 0.87)
- Offline RL with Observation Histories: Analyzing and Improving Sample Complexity (ICLR 2024 · similarity 0.87)
- Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learning (ICML 2025 · similarity 0.87)
- Horizon Reduction Makes RL Scalable (NeurIPS 2025 · similarity 0.87)
- Simple Ingredients for Offline Reinforcement Learning (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.