Research map / Reinforcement learning / Visual and representation RL
Hierarchical Reinforcement Learning with Targeted Causal Interventions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HRC, a causal hierarchical RL framework that learns a subgoal structure via targeted interventions and a new causal discovery method (SSD), with causally-guided subgoal selection rules that provably reduce training cost on trees and sparse random graphs.
Problem
Hierarchical RL with sparse rewards requires discovering the causal hierarchy among subgoals efficiently; prior causal HRL methods pick subgoals randomly for exploration and lack theoretical analysis.
Approach
Model subgoal structure as a causal graph, learn it with a tailored causal discovery method (SSD, sparse regression on an abstracted XOR-noise SCM), and prioritize which controllable subgoal to intervene on using causal-effect or A*-style ranking rules instead of uniform random selection.
Key results
- Theoretically, the targeted strategy has cost O(log^2(n) b) on trees versus Omega(n^2 b) for random selection.
- On SSD vs SDI causal discovery, SHD of 12.3 (SSD) vs 19.8 (SDI), with fewer extra edges (6.3 vs 15.6).
- HRC variants with targeted strategy learn faster than CDHRL, HAC, HER, OHRL and PPO in 2D-Minecraft (success ratio vs system probes).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SSD instead of SDI for causal discovery | HRC · Causal graph recovery | SHD: 19.8 -> 12.3 | Helped |
| Use SSD instead of SDI for causal discovery | HRC · Causal graph recovery | Missing edges: 4.2 -> 6.0 | Hurt |
| Use SSD instead of SDI for causal discovery | HRC · Causal graph recovery | Extra edges: 15.6 -> 6.3 | Helped |
| Use a targeted causal-effect ranking strategy instead of random subgoal selection | HRC · Synthetic semi-Erdős–Rényi and tree graphs | Training cost | Helped |
| Use targeted ranking with SSD instead of random selection with SSD | HRC on 2D-Minecraft · 2D-Minecraft | Success ratio | Helped |
| Use SSD instead of SDI for causal discovery | HRC with random subgoal selection · 2D-Minecraft | Success ratio | Helped |
Surprising findings
- SSD recovers slightly more missing edges (6.0 vs 4.2) than SDI, trading recall for precision.
Limitations
- Theoretical guarantees rely on assumptions such as no causal discovery errors and AND/OR-type subgoal mechanisms with binary resource variables.
- Assumes discretized, binary resource environment variables and access to all environment variables (disentangled factors).
Experimental setup
Datasets: synthetic subgoal graphs (semi-Erdos-Renyi, tree) 2D-Minecraft
Benchmarks: 2D-Minecraft synthetic subgoal graphs
Metrics: training cost (system probes) success ratio SHD (structural Hamming distance)
Baselines: CDHRL HAC HER OHRL Vanilla PPO SDI-based HRC (random strategy)
Builds on: SDI causal discovery hierarchical actor-critic A* search
Closest papers by meaning
- Causality-driven Hierarchical Structure Discovery for Reinforcement Learning (NeurIPS 2022 · similarity 0.93)
- DHRL: A Graph-Based Approach for Long-Horizon and Sparse Hierarchical Reinforcement Learning (NeurIPS 2022 · similarity 0.90)
- On Efficiency in Hierarchical Reinforcement Learning (NeurIPS 2020 · similarity 0.90)
- Causal Discovery with Reinforcement Learning (ICLR 2020 · similarity 0.90)
- Hierarchical Reinforcement Learning with Uncertainty-Guided Diffusional Subgoals (ICML 2025 · similarity 0.88)
- Probabilistic Subgoal Representations for Hierarchical Reinforcement Learning (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.