Research map / Reinforcement learning / Visual and representation RL
Prioritized Soft Q-Decomposition for Lexicographic Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes prioritized soft Q-decomposition (PSQD), which scalarizes continuous-space lexicographic multi-objective RL via a subtask transformation and incrementally learns and adapts subtask soft Q-functions, enabling zero-shot composition and offline adaptation.
Problem
Lexicographic multi-objective RL with prioritized subtasks in continuous state-action spaces, where existing methods are limited to discrete actions or trade off subtasks against priority constraints, and reuse of subtask solutions is hard.
Approach
A subtask transformation that uses the log of constraint indicator functions as transformed rewards so that lexicographic priorities become a scalar sum reward; subtask soft Q-functions are then summed and learned incrementally on-arbiter-policy, restricted to action indifference spaces of higher-priority subtasks.
Key results
- PSQD maintains zero collision cost on the top-priority obstacle-avoidance subtask while learning the lower-priority navigation subtask, whereas SAC, PPO, SQD and SQL composition do not satisfy priorities.
- In the 9-dimensional Franka Panda task, the adapted PSQD policy moves around the forbidden volume while the SQL-Comp. baseline violates the high-priority constraint with high r1 costs.
- Offline adaptation with retained replay data achieves returns comparable to the online-adapted agent and an RRT oracle in the 2D navigation task.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PSQD to adapt the lower-priority navigation subtask under lexicographic constraints | PSQD · 2D navigation | mean return | Helped |
| adapt the lower-priority navigation subtask | PSQD · 2D navigation | mean return | Helped |
| enforce lexicographic priorities during lower-priority task learning | PSQD · 2D navigation | high-priority obstacle avoidance return | Helped |
| enforce the obstacle-avoidance priority while optimizing navigation | PSQD · 2D navigation | lower-priority navigation reward | Hurt |
| use naive policy-priority extensions | SAC and PPO · 2D navigation | high-priority obstacle avoidance return | Hurt |
| replace the subtask transformation with negative rewards for priority violations (ablation) | PSQD · 2D navigation | high-priority obstacle avoidance return | Hurt |
| compose subtask Q-functions without enforcing priority constraints | SQL-Comp. · Franka Panda joint control | high-priority collision cost | Hurt |
| adapt the zero-shot prioritized solution | PSQD · Franka Panda joint control | high-priority collision cost | Helped |
| apply the priority-preserving subtask transformation to previously learned subtask solutions | PSQD · 2D navigation | obstacle collisions | Helped |
Surprising findings
- The zero-shot composed solution is suboptimal in the 2D task because it sometimes gets trapped and fails to reach the goal, even though it never collides.
Limitations
- Requires manual selection of the epsilon threshold hyperparameters, which are on the scale of subtask Q-functions and hard to estimate.
- Zero-shot composition can be suboptimal, getting stuck for some start positions.
- Offline adaptation depends on coverage of retained data for the constrained subtask MDP.
Experimental setup
Datasets: simulated 2D navigation environment data simulated Franka Emika Panda joint-control environment data
Benchmarks: 2D navigation environment Franka Emika Panda high-dimensional joint control
Metrics: mean return subtask cost (collisions) navigation cost
Baselines: SAC with priority KL objective PPO with priority KL objective soft Q-decomposition (SQD) SQL composition SQL-Comp. (sum of Q-functions) RRT oracle
Builds on: Soft Q-Learning Q-decomposition Soft Actor-Critic maximum entropy reinforcement learning
Closest papers by meaning
- Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network (ICML 2025 · similarity 0.89)
- Decoupling Meta-Reinforcement Learning with Gaussian Task Contexts and Skills (AAAI 2024 · similarity 0.88)
- Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuning (ICLR 2025 · similarity 0.88)
- XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learning (ICLR 2025 · similarity 0.88)
- Subwords as Skills: Tokenization for Sparse-Reward Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
- Progress Reward Model for Reinforcement Learning via Large Language Models (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.