Research map / Reinforcement learning / Visual and representation RL
Automaton Constrained Q-Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ACQL, a goal-conditioned constrained Q-learning method that uses a DBA from an LTL/STL spec to define subgoals and safety constraints, trained with HER and a minimum-safety classification objective.
Problem
Online RL for robotic tasks requiring temporally ordered subgoals plus time-varying (stationary and non-stationary) safety constraints specified in LTL/STL; existing LTL-RL methods have poor sparse-reward scalability and ad hoc safety handling.
Approach
Augment the MDP with the automaton state and goal lists per automaton state, use HER to densify sparse automaton-acceptance rewards, and enforce safety via a minimum-over-trajectory safety value (learned as classification with gamma_c annealed to 1) used to constrain argmax over goal-conditioned Q-values.
Key results
- ACQL achieves higher reward and success rate than LOF and CRM-RS in all five LTL task types across PointMass, Quadcopter, and Ant (e.g., 858.7 reward and 100% success on PointMass safety-reach task vs 0% for LOF and CRM-RS).
- On a 6-DOF UR5e arm with a 3-subgoal, 2-obstacle LTL task, the sim-trained policy reached 908.4 mean reward and 100% success in 16 simulated rollouts and was deployed on the real robot.
- Ablation: removing HER and using a sum-of-costs safety formulation both degrade performance on safety tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ACQL instead of LOF or CRM-RS | LTL navigation policies across PointMass, Quadcopter, and Ant · Five task types in three simulated environments | reward | Helped |
| Remove hindsight experience replay (HER) (ablation) | ACQL · LTL tasks | performance | Hurt |
| Replace minimum-safety constraints with discounted sum-of-cost constraints (ablation) | ACQL · Tasks involving safety constraints | policy quality | Hurt |
Surprising findings
- CRM-RS fails on nearly all tasks, achieving 0% success on most, indicating that rollout termination and shaping are insufficient for these LTL tasks.
- LOF achieves near-perfect success on some tasks but low reward because it does not remain near the final goal after completing the task.
Limitations
- Restricted to tasks representable by deterministic Büchi automata (recurrence class of STL/LTL).
- Real-world evaluation is a single task deployed with no partial observability or sim-to-real gap, and the sim-to-real gap is assumed negligible.
- Goal-conditioned value approximation does not capture task requirements beyond the current subgoal set.
Experimental setup
Benchmarks: PointMass LTL navigation tasks Quadcopter LTL navigation tasks Ant LTL navigation tasks UR5e cabinet real-world task
Metrics: total reward success rate
Baselines: CRM-RS LOF
Builds on: Hindsight Experience Replay Logical Options Framework Counterfactual Experiences for Reward Machines Brax JaxGCRL
Closest papers by meaning
- DeepLTL: Learning to Efficiently Satisfy Complex LTL Specifications for Multi-Task RL (ICLR 2025 · similarity 0.88)
- One Subgoal at a Time: Zero-Shot Generalization to Arbitrary Linear Temporal Logic Requirements in Multi-Task Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks (ICLR 2024 · similarity 0.88)
- Prioritized Soft Q-Decomposition for Lexicographic Reinforcement Learning (ICLR 2024 · similarity 0.87)
- GoalLadder: Incremental Goal Discovery with Vision-Language Models (NeurIPS 2025 · similarity 0.87)
- Zero-Shot Instruction Following in RL via Structured LTL Representations (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.