Research map / Reinforcement learning / Policy gradient and actor-critic
Coarse-to-fine Q-Network with Action Sequence for Data-Efficient Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CQN-AS, a critic-only value-based RL method whose coarse-to-fine critic outputs Q-values over action sequences, improving CQN on sparse-reward BiGym and RLBench tasks and HumanoidBench locomotion.
Problem
Value-based RL for continuous robotic control with sparse rewards struggles to learn long-horizon value functions; actor-critic methods with action sequences suffer value overestimation in high-dimensional action spaces.
Approach
Train a coarse-to-fine critic (CQN) to output Q-values for every step of an action sequence in parallel, aggregating per-step features with a GRU; because the critic-only design has no separate actor to exploit value errors, learning with action sequences stays stable.
Key results
- On 25 BiGym tasks, CQN-AS matches ACT quickly and outperforms it through online learning on most tasks, while other RL baselines get 0% on many tasks.
- On 20 RLBench tasks, CQN-AS achieves similar performance to CQN on most tasks, with large gains on long-horizon tasks such as Open Oven.
- On HumanoidBench, CQN-AS significantly improves over CQN and outperforms SAC on most of the 8 locomotion tasks, with larger gains on Walk and Run.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use action sequences instead of single-step actions for return-to-go prediction | return-to-go prediction model · BiGym humanoid demonstrations | validation L1 loss | Helped |
| use CQN-AS with action sequences instead of CQN with single-step actions | CQN · BiGym | success rate | Helped |
| use CQN-AS with action sequences instead of CQN with single-step actions | CQN · RLBench | success rate | Mixed |
| use CQN-AS with action sequences instead of CQN | CQN · HumanoidBench | episode return | Helped |
| combine CQN-AS with Cal-QL instead of CQN with Cal-QL | Cal-QL on BiGym Sandwich Remove · BiGym Sandwich Remove | success rate: 7 ($\pm$ 14) \% -> 33 ($\pm$ 6.8) \% | Helped |
| increase action sequence length (ablation) | CQN-AS · robotic tasks | performance | Mixed |
| remove the RL objective and train only with BC on successful demonstrations (ablation) | CQN-AS · robotic tasks | performance | Hurt |
| use too high an N-step return (ablation) | CQN-AS · robotic tasks | performance | Hurt |
| remove temporal ensemble (ablation) | CQN-AS · Saucepan To Hob | performance | Hurt |
| add temporal ensemble (ablation) | CQN-AS · Reach Target Single | performance | Hurt |
| increase temporal ensemble magnitude (ablation) | CQN-AS · Reach Target Single | performance | Helped |
| increase temporal ensemble magnitude (ablation) | CQN-AS · Saucepan To Hob | performance | Hurt |
Surprising findings
- SAC and TD3 with action sequences suffer severe value overestimation and fail on the HumanoidBench stand task.
- CQN-AS underperforms CQN on DeepMind Control locomotion tasks with torque control.
Limitations
- No real-world robot evaluation.
- Struggles on long-horizon tasks requiring delicate object interaction such as cups or cutlery.
- Torque-control locomotion tasks show degraded performance relative to CQN.
Experimental setup
Datasets: BiGym demonstrations RLBench motion-planned demonstrations HumanoidBench
Benchmarks: BiGym RLBench HumanoidBench DeepMind Control Suite
Metrics: success rate episode return validation L1 loss
Baselines: CQN DrQ-v2+ ACT SAC DreamerV3 TD-MPC2
Builds on: Coarse-to-fine Q-Network (CQN) Action Chunking with Transformers (ACT) DrQ-v2 Dueling network Cal-QL
Closest papers by meaning
- Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network (ICML 2025 · similarity 0.88)
- Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuning (ICLR 2025 · similarity 0.87)
- Adaptive $Q$-Network: On-the-fly Target Selection for Deep Reinforcement Learning (ICLR 2025 · similarity 0.87)
- Stochastic Q-learning for Large Discrete Action Spaces (ICML 2024 · similarity 0.87)
- Reinforcement Learning with Action Chunking (NeurIPS 2025 · similarity 0.87)
- Deep Implicit Imitation Reinforcement Learning in Heterogeneous Action Settings (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.