Research map / Reinforcement learning / Offline RL
Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BOSA, a cross-domain offline RL method that uses support-constrained policy and value optimization (with a CVAE-based behavior model and transition-model filter) to handle both OOD state-actions and OOD transition dynamics when mixing limited target data with source-domain data.
Problem
Offline RL is data-inefficient and suffers from OOD state-actions; when source-domain data from different transition dynamics is added, naive mixing often hurts target performance due to OOD transition dynamics.
Approach
Identify OOD transition dynamics as a distinct failure of cross-domain offline RL and address it with a supported value optimization that only bootstraps on source transitions whose estimated target-domain transition likelihood exceeds a threshold, combined with a supported policy optimization (CVAE-estimated behavior policy constraint via Lagrangian) and a conservative value regularizer.</いcore_i
Key results
- With only 10% of target D4RL data plus source data, BOSA incurs the smallest average drop (-25.6%) relative to best 100%-data baselines, versus -50.1% to -92.5% for other baselines.
- BOSA outperforms DARA on the summed cross-domain body-mass tasks (415.3 vs 383.2) and joint-noise tasks (417.6 vs 413.1) in the table comparison.
- BOSA improves over the SPOT (10%) baseline in 11 of 12 tasks when source data is generated by pseudo-transition model or noise augmentation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BOSA with 10% target data and cross-domain source data instead of the strongest 100% target-data baseline | Offline RL on D4RL cross-domain tasks · 18 D4RL tasks | average performance degradation: -25.6% | Helped |
| Add source-domain data without BOSA's support constraints | BCQ, MOPO, CQL, and SPOT · D4RL cross-domain tasks | normalized average score | No clear effect |
| Use BOSA rather than SPOT with generated source data | Offline RL with pseudo-model or noise augmentation · 12 cross-domain tasks | normalized average score | Helped |
| Remove supported policy regularization (ablation) | BOSA · Cross-domain offline RL tasks | performance | Hurt |
| Remove the transition filter from value optimization (ablation) | BOSA · Cross-domain offline RL tasks | performance: around 20% | Hurt |
| Remove conservation regularization from value optimization (ablation) | BOSA · Cross-domain offline RL tasks | performance | Hurt |
| Replace filtered mixed-domain value optimization with target-domain-only value optimization (ablation) | BOSA · Cross-domain offline RL tasks | performance | Hurt |
| Increase the target-domain data amount (ablation) | BOSA and SPOT · Cross-domain D4RL tasks | normalized average score | Helped |
| Increase the transition-model ensemble size (ablation) | BOSA · Cross-domain offline RL tasks | performance | Helped |
Surprising findings
- Naively adding cross-domain source data often fails to improve and sometimes degrades performance (e.g., CQL in 6 of 18 tasks).
Limitations
- Assumes source and target share state space, action space, and reward function.
- Evaluated mainly on simulated MuJoCo tasks and one simulated quadruped; no real-world robot experiments.
Experimental setup
Datasets: D4RL Hopper D4RL Walker2d D4RL HalfCheetah DARA-style source data (body mass / joint noise shift) quadruped robot simulation (A1)
Benchmarks: D4RL Hopper D4RL Walker2d D4RL HalfCheetah quadruped robot simulation
Metrics: normalized average return average performance change (%)
Baselines: BCQ MOPO CQL SPOT DARA MABE Finetune
Builds on: BCQ CQL MOPO SPOT DARA TD3
Closest papers by meaning
- Localized Dynamics-Aware Domain Adaption for Off-Dynamics Offline Reinforcement Learning (NeurIPS 2026 · similarity 0.93)
- Offline Reinforcement Learning with OOD State Correction and OOD Action Suppression (NeurIPS 2024 · similarity 0.92)
- ODRL: A Benchmark for Off-Dynamics Reinforcement Learning (NeurIPS 2024 · similarity 0.92)
- Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL (ICML 2024 · similarity 0.91)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- Hybrid Reinforcement Learning from Offline Observation Alone (ICML 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.