Research map / Reinforcement learning / Imitation learning
Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ARSQ, a value-based soft Q-learning method that decomposes continuous actions coarse-to-fine and predicts dimensional soft advantages auto-regressively, improving learning from suboptimal offline and online-collected demonstrations.
Problem
Value-based RL for continuous control with offline demonstrations and suboptimal data; independent per-dimension Q estimation ignores cross-dimension dependencies and biases action selection toward frequent suboptimal behaviors.
Approach
Extends soft Q-learning with dimensional soft advantages that are normalized per dimension and sum to the joint soft advantage, so the joint action is generated auto-regressively across dimensions and coarse-to-fine levels, combined with a behavior-cloning margin loss on demonstrations.
Key results
- On D4RL, ARSQ achieves an average 1.62x performance improvement over SOTA value-based baseline CQN when trained with suboptimal demonstrations.
- In the fully offline D4RL setting, ARSQ reaches a total of 741.1 across nine datasets, above the best baseline CQL at 698.5.
- On hopper-medium, ARSQ scores 99.2 ± 0.5 versus 67.6 for the next-best offline baseline (DT) in the fully offline table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use auto-regressive action decomposition instead of independent action-dimension Q estimation | ARSQ versus value-based baselines · D4RL | performance: 1.62× | Helped |
| reverse the conditioning order (dimensional before coarse-to-fine) (ablation) | ARSQ · D4RL hopper-medium-expert, hopper-medium-replay; RLBench Open Oven | performance | Hurt |
| remove coarse-to-fine conditioning (ablation) | ARSQ · D4RL hopper-medium-expert, hopper-medium-replay; RLBench Open Oven | performance | Hurt |
| remove dimensional conditioning (ablation) | ARSQ · D4RL hopper-medium-expert, hopper-medium-replay; RLBench Open Oven | performance | Hurt |
| replace coarse-to-fine discretization with B^L bins per action dimension (ablation) | ARSQ · D4RL hopper-medium-expert, hopper-medium-replay; RLBench Open Oven | performance | Hurt |
| remove coarse-to-fine structure and dimensional conditioning (ablation) | ARSQ · D4RL hopper-medium-expert, hopper-medium-replay; RLBench Open Oven | performance | Hurt |
| use separate networks for each action dimension instead of a shared backbone (ablation) | ARSQ advantage network · D4RL hopper-medium; RLBench Open Oven | performance | Hurt |
| use level-shared networks instead of a shared backbone (ablation) | ARSQ advantage network · D4RL hopper-medium; RLBench Open Oven | performance | Hurt |
| use a very high temperature coefficient (ablation) | ARSQ · D4RL hopper-medium; RLBench Open Oven | performance | Hurt |
| use a very low temperature coefficient (ablation) | ARSQ · D4RL hopper-medium; RLBench Open Oven | policy improvement | Hurt |
| remove coarse-to-fine action discretization (ablation) | ARSQ · Q prediction case study | Q prediction error: 0.16 ± 0.02 -> 0.50 ± 0.21 | Hurt |
| use the variant behavior-cloning objective (ablation) | ARSQ · D4RL medium and medium-expert datasets | performance | Helped |
Surprising findings
- Independent per-dimension Q decomposition yields a Q prediction error of 17.57 versus 0.16 for ARSQ on a toy one-step task.
- ARSQ has higher inference latency than CQN (32.1 ms vs 6.9 ms on RLBench), a stated limitation.
Limitations
- Higher inference latency than CQN because dimensions are generated sequentially
- Coarse-to-fine discretization uses fixed bins and levels; adaptive discretization is left to future work
- Performance on RLBench reported on six selected tasks of 20 in the main text
Experimental setup
Datasets: D4RL halfcheetah, hopper, walker2d (medium, medium-replay, medium-expert) RLBench (100 expert demonstrations per task)
Benchmarks: D4RL RLBench
Metrics: normalized return success rate total normalized score
Baselines: CQN DrQ-v2 DrQ-v2+ ACT BC CQL IQL TD3+BC Onestep RL RvS-R Filtered BC Decision Transformer
Builds on: soft Q-learning CQN DrQ-v2 DQN
Closest papers by meaning
- Actor-Free Continuous Control via Structurally Maximizable Q-Functions (NeurIPS 2025 · similarity 0.89)
- Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control (ICLR 2025 · similarity 0.89)
- Prioritized Soft Q-Decomposition for Lexicographic Reinforcement Learning (ICLR 2024 · similarity 0.89)
- Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learning (ICLR 2025 · similarity 0.88)
- Adaptive $Q$-Network: On-the-fly Target Selection for Deep Reinforcement Learning (ICLR 2025 · similarity 0.88)
- Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.