Research map / Reinforcement learning / Visual and representation RL
Towards Generalizable Reinforcement Learning via Causality-Guided Self-Adaptive Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CSR, a causality-guided world-model approach that detects whether environment changes are distribution shifts or state/action space expansions and adapts a causal model with few samples for policy transfer on CartPole, CoinRun and Atari.
Problem
Generalizing RL policies when both the distribution and the environment (state/action) spaces change between source and target tasks, which prior policy-adaptation methods typically ignore.
Approach
Augment a Dreamer-style world model with causal structural masks and a task-specific change factor; detect whether a new task needs only the change factor re-estimated or new latent causal variables added (with a self-adaptive RL-based expansion policy), then prune non-compact variables via sparsity regularization.
Key results
- On CartPole, CSR reaches the maximum score of 500 on all four tasks, while Dreamer and AdaRL drop to 311.6 and 407.5 on Task 3 and Task 4 respectively.
- On five Atari games, CSR achieves the highest mean score in all five, e.g., Crazy Climber 88306.5 vs. AdaRL 62565.3 and Dreamer 68026.5.
- Self-Adaptive expansion yields fewer expanded causal variables than Random or Deterministic expansion on CartPole (4.0 vs 6.2 and 6.0).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CSR instead of AdaRL for adaptation across CartPole tasks | CartPole agents · CartPole Tasks 1–4 | score: Task 1: 500.0 ($\pm$ 0.0) vs 500.0 ($\pm$ 0.0); Task 2: 500.0 ($\pm$ 0.0) vs 468.0 ($\pm$ 2.3); Task 3: 500.0 ($\pm$ 0.0 | Helped |
| use CSR instead of the strongest listed baseline | Atari agents · Alien, Bank Heist, Crazy Climber, Gopher, Pong | score: Alien: 1147.5 -> 1586.9; Bank Heist: 1313.7 -> 1454.1; Crazy Climber: 68026.5 -> 88306.5; Gopher: 5607.3 -> 6718.6; Pong | Helped |
| add structural matrices D to the world model (ablation) | CSR · Atari games | cumulative reward | Helped |
| search for the expansion size with the Self-Adaptive strategy rather than use random or deterministic expansion (ablation) | CSR · Simulated environments, CartPole, CoinRun, Atari | expansion performance | Helped |
| use the Self-Adaptive expansion strategy (ablation) | CSR · Simulated, CartPole, CoinRun, Atari | added causal variables: Simulated: Random 4.0, Deterministic 5.0, Self-Adaptive 4.2; CartPole: Random 6.2, Deterministic 6.0, Self-Adaptive 4.0; | Mixed |
Limitations
- Only considers domain generalization and not nonstationary changes over time.
- Identifiability of the transition change factor is proven only in linear cases; nonlinear case is supported empirically only.
- Self-adaptive expansion search requires extensive training time for models of different scales.
Experimental setup
Datasets: CartPole (simulated variants) CoinRun Atari 100K
Benchmarks: CartPole CoinRun Atari 100K (Alien, Bank Heist, Crazy Climber, Gopher, Pong)
Metrics: episode return / score minimum adaptation steps
Baselines: DQN SPR Dreamer AdaRL EfficientZero
Builds on: Dreamer AdaRL DEN EfficientZero
Closest papers by meaning
- Tackling Non-Stationarity in Reinforcement Learning via Causal-Origin Representation (ICML 2024 · similarity 0.88)
- Modeling Unseen Environments with Language-guided Composable Causal Components in Reinforcement Learning (ICLR 2025 · similarity 0.88)
- Learning Generalizable and Composable Abstractions for Transfer in Reinforcement Learning (AAAI 2024 · similarity 0.87)
- Causality Meets Locality: Provably Generalizable and Scalable Policy Learning for Networked Systems (NeurIPS 2025 · similarity 0.87)
- Learning Time-Aware Causal Representation for Model Generalization in Evolving Domains (ICML 2025 · similarity 0.87)
- Fine-Grained Causal Dynamics Learning with Quantization for Improving Robustness in Reinforcement Learning (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.