Research map / Reinforcement learning / Visual and representation RL
Tackling Non-Stationarity in Reinforcement Learning via Causal-Origin Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes COREP, a dual graph-attention representation (core-GAT guided by TD-error detection plus general-GAT, with a VAE) that learns a stable causal-origin graph state representation for PPO to handle non-stationary RL.
Problem
Reinforcement learning under complex non-stationary dynamics, where existing methods must explicitly model environment changes and often need prior knowledge of the environment.
Approach
Treats non-stationarity as propagating through causal relationships among state elements and learns an environment-shared union graph (a MAG) via a TD-error-guided core graph attention network plus a continuously updated general graph attention network, feeding the resulting latent into PPO without explicit change detection.
Key results
- COREP outperforms PPO, FN-VAE and VariBAD across DMC tasks under periodic and randomized non-stationarity, with a narrow gap to the Oracle (5 seeds).
- Ablations show removing the guided update (w/o L_guide), the single-GAT design, or the MAG/sparsity regularizers degrades returns.
- COREP's performance stays consistent across non-stationarity degrees alpha_d while baselines degrade more strongly.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use COREP instead of FN-VAE, VariBAD, or PPO | DeepMind Control Suite agents · DeepMind Control Suite (W+A)-EP | return | Helped |
| Add the COREP-specific designs to a VAE-only agent (ablation) | COREP · DeepMind Control Suite (W+A)-EP, W-EP, and A-EP | normalized return | Helped |
| Add a second GAT and its corresponding update mechanism (ablation) | COREP · DeepMind Control Suite (W+A)-EP, W-EP, and A-EP | normalized return | Helped |
| Add TD error detection and the guided update mechanism (ablation) | COREP · DeepMind Control Suite (W+A)-EP, W-EP, and A-EP | normalized return | Helped |
| Add the sparsity regularization loss (ablation) | COREP · DeepMind Control Suite (W+A)-EP, W-EP, and A-EP | normalized return | Helped |
| Add the MAG regularization loss (ablation) | COREP · DeepMind Control Suite (W+A)-EP, W-EP, and A-EP | normalized return | Helped |
| Add the VAE process (ablation) | COREP · DeepMind Control Suite (W+A)-EP, W-EP, and A-EP | normalized return | Helped |
| Vary the degree of non-stationarity | COREP · Cheetah Run and Swimmer Swimmer6 | normalized return | No clear effect |
Limitations
- Scalability to high-dimensional state spaces due to computationally intensive graph-based representation.
- Theoretical analysis uses deterministic dynamics and simplified assumptions; sub-environment decomposition is only approximate in practice.
Experimental setup
Datasets: DeepMind Control Suite
Benchmarks: DeepMind Control Suite
Metrics: episode return normalized mean return variance across seeds
Baselines: FN-VAE VariBAD PPO Oracle
Builds on: PPO FN-VAE maximal ancestral graphs graph attention network VariBAD
Closest papers by meaning
- Towards Generalizable Reinforcement Learning via Causality-Guided Self-Adaptive Representations (ICLR 2025 · similarity 0.88)
- Fine-Grained Causal Dynamics Learning with Quantization for Improving Robustness in Reinforcement Learning (ICML 2024 · similarity 0.87)
- Sim-to-Real Causal Transfer: A Metric Learning Approach to Causally-Aware Interaction Representations (CVPR 2025 · similarity 0.87)
- Causal Information Prioritization for Efficient Reinforcement Learning (ICLR 2025 · similarity 0.86)
- Hierarchical Reinforcement Learning with Targeted Causal Interventions (ICML 2025 · similarity 0.86)
- Curious Causality-Seeking Agents Learn Meta Causal World (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.