Research map / Reinforcement learning / Policy gradient and actor-critic

Run-Time Task Composition with Safety Semantics

ICML 2024 · Reinforcement learning › Policy gradient and actor-critic · approach: Safe policy improvement

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Extends Boolean task composition (Nangue Tasse et al. 2020) with two safety-aware negation semantics (minimum-violation and prioritized safety), proves correctness under assumptions, and extends zero-shot composition to continuous action spaces.

Problem

Existing Boolean task composition in RL only supports reachability-only negation and discrete actions, so it cannot express zero-shot safety/avoidance constraints or compose policies in continuous action spaces.

Approach

Penalty-hierarchy reward shaping (multiplier eta) for positive and negated tasks yields minimum-violation and prioritized-safety paths; extended Q-value max/min/negation operators compose these policies at run time, and disjunction/conjunction are derived analytically over continuous-action policies via argmax selection between component policies.

Key results

• Composed policies via zero-shot Boolean operators produce pure paths when possible and MV or PS paths otherwise, shown qualitatively across three environments.
• Theorems 1-6 give correctness of MV and PS semantics and composition under stated assumptions, with PS composition requiring Assumption 4.1 to avoid chattering.
• First demonstration of Boolean task composition in a continuous action space (Bullet-Safety-Gym with TD3).

Measured effects reported in the paper

Surprising findings

• Naive composition of prioritized safety negated policies can cause chattering (infinite loops) in some constructed environments.
• Penalty multiplier eta from theory can be very large; in practice smaller values suffice and conservative values slow convergence.

Limitations

• PS composition lacks formal guarantees without Assumption 4.1 and can chatter in general.
• Correctness proofs are for optimal policies in deterministic MDPs; function approximation is only demonstrated empirically.
• Multiple learning rates and curriculum tuning needed for function-approximation experiments.

Experimental setup

Datasets: grid world environments (value iteration) item collection grid world with image observations Bullet-Safety-Gym

Benchmarks: grid world (value iteration) item collection grid world with image observations Bullet-Safety-Gym

Metrics: qualitative path comparison policy value and path behavior

Baselines: Nangue Tasse et al. 2020 reachability-only composition

Builds on: Nangue Tasse et al. 2020 Boolean task algebra Q-learning DQN TD3 value iteration

Closest papers by meaning

• Compositional Behavioral Semantics for State Abstraction in Reinforcement Learning (ICML 2026 · similarity 0.88)
• Compositional Reinforcement Learning from Logical Specifications (NeurIPS 2021 · similarity 0.87)
• Compositional Reasoning in Language Models under Reinforcement Learning Post-Training (NeurIPS 2026 · similarity 0.85)
• Safeguarded Progress in Reinforcement Learning: Safe Bayesian Exploration for Control Policy Synthesis (AAAI 2024 · similarity 0.85)
• Modeling Unseen Environments with Language-guided Composable Causal Components in Reinforcement Learning (ICLR 2025 · similarity 0.84)
• Compositional Automata Embeddings for Goal-Conditioned Reinforcement Learning (NeurIPS 2024 · similarity 0.84)