Research map / Reinforcement learning / Visual and representation RL
An Offline Adaptation Framework for Constrained Multi-Objective Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PDOA, an offline adaptation framework that infers target preferences (and conservative safety-weighted preferences) from a few demonstrations on top of pretrained offline MORL policies, without handcrafted preferences or safety thresholds.
Problem
Offline multi-objective RL and constrained MORL usually require handcrafted target preferences and safety thresholds at deployment, which are hard to specify; the paper asks how to infer them from a few expert demonstrations.
Approach
Learn a preference-conditioned policy set offline (MODF or PEDA), then fit a Gaussian posterior over preferences to a few demonstrations by maximizing a surrogate likelihood combining a TD-error reward, action likelihood, and a prior; constraints are handled by converting to an unconstrained problem with an extended preference over costs and applying a CVaR-based conservative weight on cost object
Key results
- PDOA[MODF] matches the oracle performance of preference-informed methods on D4MORL Amateur datasets and yields broader Pareto fronts than BC-Finetune and Prompt-MODT.
- On DSRL safe RL tasks, PDOA[MODF] achieves relatively safe performance closely matching the threshold-informed CDT oracle, with reward comparable to or exceeding CDT.
- PDOA[MORvS] performs poorly on DSRL datasets because MORvS needs accurate return predictions for each preference.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PDOA [MODF] to adapt preferences from demonstrations | PDOA [MODF] · D4MORL Amateur | average utility | Helped |
| Use PDOA [MODF] to adapt preferences from demonstrations | PDOA [MODF] · D4MORL Amateur | Hypervolume | Helped |
| Adapt target preferences from demonstrations rather than behavior-cloning fine-tuning or prompting | D4MORL Amateur · D4MORL Amateur | Hypervolume | Helped |
| Use PDOA [MODF] for adaptation without known safety thresholds | PDOA [MODF] · DSRL safe RL tasks | constraint satisfaction | Helped |
| Use PDOA [MODF] rather than threshold-informed CDT | DSRL safe RL tasks · DSRL safe RL tasks | reward | Mixed |
| Use PDOA [MORvS] as the policy-learning instance | PDOA · DSRL safe RL tasks | reward | Hurt |
| Use PDOA [MODF] to adapt preferences and safety weights from demonstrations | PDOA [MODF] · CMORL tasks | average utility | Helped |
| Increase demonstration size (ablation) | PDOA [MODF] · MORL tasks | average utility | Helped |
| Increase demonstration size (ablation) | PDOA [MODF] · MORL tasks | Hypervolume | Helped |
| Increase demonstration size (ablation) | PDOA [MODF] · Safe RL tasks | reward | Helped |
| Increase conservatism parameter alpha (ablation) | PDOA [MODF] · Safe RL tasks | constraint violations | Helped |
Surprising findings
- PDOA[MORvS] performs poorly on DSRL datasets, attributed to abundance of suboptimal trajectories.
Limitations
- Adaptation involves multiple preference sampling and gradient updates, adding computational burden and latency at deployment.
- Constructing demonstrations may still require human expertise.
Experimental setup
Datasets: D4MORL DSRL CMO datasets (constructed)
Benchmarks: D4MORL DSRL CMORL (CMO-Hopper, CMO-HalfCheetah, CMO-Ant, CMO-Swimmer)
Metrics: average utility hypervolume maximum cost return
Baselines: BC-Finetune Prompt-MODT MODF MORvS CDT
Builds on: Diffusion-QL MODF PEDA (MORvS) Prompt-DT CDT
Closest papers by meaning
- Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning (AAAI 2025 · similarity 0.90)
- From Static Policies to Adaptive Priors in Offline Reinforcement Learning (NeurIPS 2026 · similarity 0.89)
- Online Optimization for Offline Safe Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Provable Offline Preference-Based Reinforcement Learning (ICLR 2024 · similarity 0.88)
- The Max-Min Formulation of Multi-Objective Reinforcement Learning: From Theory to a Model-Free Algorithm (ICML 2024 · similarity 0.88)
- Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.