Research map / Reinforcement learning / Visual and representation RL
One Subgoal at a Time: Zero-Shot Generalization to Arbitrary Linear Temporal Logic Requirements in Multi-Task Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GenZ-LTL, which decomposes LTL specifications via Büchi automata into reach-avoid subgoals, solves them one at a time with a state-wise HJ-reachability safe RL formulation plus subgoal-induced observation reduction, and achieves zero-shot generalization to unseen LTL specs.
Problem
Zero-shot generalization of RL policies to arbitrary LTL task specifications with nested long-horizon objectives and safety constraints, including handling unsatisfiable subgoals and the exponential blowup of observation-subgoal combinations.
Approach
Solve Büchi-automaton-derived reach-avoid subgoals one at a time using a goal-conditioned policy trained with state-wise HJ-reachability safety constraints (Lagrangian), reduce observations by subgoal-induced symmetry (reach vs. fused avoid channels), and select subgoals at test time via learned value functions with a timeout-based switching mechanism.
Key results
- On LetterWorld finite-horizon specs, GenZ-LTL achieves 0.95 average success rate with 0.00 violation rate, versus 0.76 for DeepLTL and 0.82 for RAD-embeddings.
- On ZoneEnv finite-horizon specs, GenZ-LTL achieves 0.98 average success rate with 0.01 violation rate, versus 0.87 for DeepLTL.
- On infinite-horizon ZoneEnv specs, GenZ-LTL reduces violation rates to 0.03-0.07 while DeepLTL and GCRL-LTL show 0.13-0.40.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use one-subgoal-at-a-time reach-avoid policies with state-wise reachability constraints and observation reduction instead of baseline approaches | zero-shot LTL generalization · LetterWorld φ1–φ8 and ZoneEnv φ9–φ16 | success rate: 0.95_{\pm0.03} vs 0.76_{\pm0.05}; 0.98_{\pm0.02} vs 0.87_{\pm0.08} | Helped |
| Use state-wise reachability constraints to enforce safety | GenZ-LTL · infinite-horizon LetterWorld ψ1–ψ3 | violation rate: 0.00_{\pm0.00} vs 0.09_{\pm0.04} | Helped |
| Add subgoal-induced observation reduction (ablation) | GenZ-LTL · ZoneEnv reach-avoid tasks, n=2 | success rate | Helped |
| Add state-wise constraints (ablation) | GenZ-LTL · ZoneEnv reach-avoid tasks, n=2 | safety performance | Helped |
| Increase the number of subgoals in the specification | GenZ-LTL · ZoneEnv reach-only and reach-avoid specifications | success rate | Mixed |
Surprising findings
- Solving subgoals one at a time (described as myopic) outperforms conditioning on full subgoal sequences or the whole automaton for zero-shot generalization.
- Baselines that model no explicit safety constraint achieve substantially higher violation rates even when success is reasonable.
Limitations
- Observation reduction relies on assumed proposition-independent/dependent decomposition and coordinate-wise monotone observations, which are environment-specific.
- No guarantee that a satisfiable LTL specification is solved, since greedy sequential subgoal solving can fail; failure does not imply unsatisfiability.
- Evaluation limited to navigation-style grid and point-robot environments.
Experimental setup
Datasets: LetterWorld (grid world, custom) ZoneEnv (continuous navigation, adapted from prior work)
Benchmarks: LetterWorld finite-horizon LTL specs ZoneEnv finite-horizon LTL specs infinite-horizon LTL specs ZoneEnv subgoal-chaining and environment-complexity variants
Metrics: success rate violation rate average steps to satisfy average visits to accepting states
Baselines: LTL2Action GCRL-LTL DeepLTL RAD-embeddings
Builds on: DeepLTL LTL2Action GCRL-LTL RAD-embeddings PPO-Lagrangian
Closest papers by meaning
- DeepLTL: Learning to Efficiently Satisfy Complex LTL Specifications for Multi-Task RL (ICLR 2025 · similarity 0.93)
- Zero-Shot Instruction Following in RL via Structured LTL Representations (NeurIPS 2026 · similarity 0.91)
- Zero Shot Generalization of Vision-Based RL Without Data Augmentation (ICML 2025 · similarity 0.88)
- Automaton Constrained Q-Learning (NeurIPS 2025 · similarity 0.88)
- Reinforcement Learning with LTL and $\omega$-Regular Objectives via Optimality-Preserving Translation to Average Rewards (NeurIPS 2024 · similarity 0.88)
- Foundation Policies with Hilbert Representations (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.