Research map / Reinforcement learning / Visual and representation RL
Causal Information Prioritization for Efficient Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CIP, an RL method that uses factored-MDP causal discovery (DirectLiNGAM) to counterfactually augment state transitions and reweight actions, plus a causality-aware empowerment objective, evaluated on 39 continuous-control tasks.
Problem
Sample inefficiency of RL caused by blind exploration that ignores reward-relevant causal relationships among states, actions and rewards.
Approach
Learns state-to-reward and action-to-reward causal masks with DirectLiNGAM in a factored MDP; swaps reward-irrelevant state features across transitions for counterfactual augmentation, reweights actions by causal weights, and adds a causal empowerment (mutual information) term to the maximum-entropy actor-critic objective.
Key results
- On the 8 locomotion tasks, CIP achieves the best score on several tasks, e.g., 12594±210 on HalfCheetah vs. 10888 for SAC and 9390 for ACE.
- In Adroit Hand door open (28-DoF action), CIP reaches near 100% success after 700k steps while the other compared methods show no significant progress.
- CIP reports near-perfect scores of 100 on 17 Meta-World robot-arm tasks in the normalized score comparison.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add counterfactual data augmentation (ablation) | CIP · Ant | average return: 6231±81 -> 6418±81 | Helped |
| Add counterfactual data augmentation (ablation) | CIP · HalfCheetah | average return: 12225±102 -> 12594±210 | Helped |
| Add counterfactual data augmentation (ablation) | CIP · Hopper | average return: 2308±785 -> 2846±882 | Helped |
| Add counterfactual data augmentation (ablation) | CIP · Walker2d | average return: 5294±41 -> 5624±91 | Helped |
| Add counterfactual data augmentation (ablation) | CIP · Cheetah Run | average return: 885±13 -> 893±12 | Helped |
| Add counterfactual data augmentation (ablation) | CIP · Hopper Stand | average return: 931±22 -> 936±17 | Helped |
| Add counterfactual data augmentation (ablation) | CIP · Quadruped Walk | average return: 945±35 -> 948±54 | Helped |
| Add counterfactual data augmentation (ablation) | CIP · Reacher Hard | average return: 989±13 -> 991±11 | Helped |
| Add causal action empowerment (ablation) | CIP · Ant | average return: 6295±210 -> 6418±81 | Helped |
| Add causal action empowerment (ablation) | CIP · HalfCheetah | average return: 10986±572 -> 12594±210 | Helped |
| Add causal action empowerment (ablation) | CIP · Hopper | average return: 2270±904 -> 2846±882 | Helped |
| Add causal action empowerment (ablation) | CIP · Walker2d | average return: 5547±91 -> 5624±91 | Helped |
Surprising findings
- Without counterfactual augmentation, CIP still reaches 100% success on Meta-World tasks when using causally-informed states, and the w/o-Emp variant underperforms on several locomotion tasks.
Limitations
- Not yet extended to complex real-world 3D robotics tasks.
- Does not account for non-stationarity and heterogeneity in causal discovery.
- Relies on DirectLiNGAM for causal discovery; alternatives not explored.
Experimental setup
Datasets: MuJoCo DMControl Meta-World Adroit Hand Cartpole
Benchmarks: MuJoCo DMControl Meta-World Adroit Hand Cartpole
Metrics: average return success rate normalized score optimality gap
Baselines: SAC ACE BAC IFactor
Builds on: SAC ACE DirectLiNGAM IFactor
Closest papers by meaning
- Hierarchical Reinforcement Learning with Targeted Causal Interventions (ICML 2025 · similarity 0.88)
- CIG: Exploration via Conditional Information Gain (NeurIPS 2026 · similarity 0.87)
- ACAMDA: Improving Data Efficiency in Reinforcement Learning through Guided Counterfactual Data Augmentation (AAAI 2024 · similarity 0.87)
- Tackling Non-Stationarity in Reinforcement Learning via Causal-Origin Representation (ICML 2024 · similarity 0.86)
- Building Minimal and Reusable Causal State Abstractions for Reinforcement Learning (AAAI 2024 · similarity 0.86)
- Exploiting Causal Graph Priors with Posterior Sampling for Reinforcement Learning (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.