Research map / Reinforcement learning / Policy gradient and actor-critic
Causality Meets Locality: Provably Generalizable and Scalable Policy Learning for Networked Systems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GSAC, which combines causal mask recovery (approximately compact representations) with meta actor-critic training to make networked MARL policies scalable and generalizable to new domains, with finite-sample guarantees.
Problem
Networked multi-agent RL (e.g., traffic, power, wireless grids) suffers from exponential growth in joint state-action space with agent count and from environment/domain shifts that require fast adaptation.
Approach
Each agent learns a sparse local causal mask to identify minimal neighborhood variables influencing its dynamics, building approximately compact representations of states and domain factors; a meta actor-critic trains a shared domain-conditioned policy across source domains and adapts at test time by estimating the new domain factor from a few trajectories.
Key results
- GSAC achieves the best few-shot adaptation performance (episodes 1-30) on wireless benchmarks across grid sizes 3, 4, and 5 (16 to 36 agents), relative to SAC-MTL, SAC-FT, and SAC-LFS.
- Theory gives an ACR approximation error of 3 r̄/(1-γ) γ^(κ+1) and finite-sample bounds for critic, actor, and adaptation gap, with O(1/sqrt(T_a)) adaptation rate.
- Experiments use only 3 source domains and a target domain factor of 0.65, with 20 trajectories per domain for factor estimation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Condition the policy on estimated domain factors after meta-training across source domains | GSAC · Wireless communication benchmark, grid sizes 3, 4, and 5 | return | Helped |
| Use domain-factor conditioning instead of pre-specified one-hot domain labels | SAC-MTL · Wireless communication benchmark | return | Helped |
| Meta-train across source domains before target-domain learning | SAC-LFS · Wireless communication benchmark | return | Helped |
| Estimate target domain factors and deploy without further policy training | GSAC · Wireless communication benchmark, Episodes 1–30 | few-shot return | Helped |
| Fine-tune policy parameters directly in the target domain rather than deploy the meta-policy immediately | SAC-FT · Wireless communication benchmark, early adaptation | return | Hurt |
Limitations
- Evaluated only on tabular and fully observed benchmarks.
- Continuous state/action spaces and partial observability are not addressed.
- Experiments are small-scale with a few source domains (M=3).
Experimental setup
Datasets: wireless communication benchmark traffic control benchmark
Benchmarks: wireless communication networks (grid sizes 3, 4, 5) traffic control
Metrics: episodic return few-shot adaptation performance
Baselines: SAC-MTL SAC-FT SAC-LFS
Builds on: exponential decay property for networked MARL SAC AdaRL PEARL
Closest papers by meaning
- Towards Generalizable Reinforcement Learning via Causality-Guided Self-Adaptive Representations (ICLR 2025 · similarity 0.87)
- Bayesian Ego-graph inference for Networked Multi-Agent Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
- Scalable Multi-Agent Reinforcement Learning for Networked Systems with Average Reward (NeurIPS 2020 · similarity 0.87)
- Generative Actor-Critic with Soft Bridge Policies (NeurIPS 2026 · similarity 0.86)
- ACE: Off-Policy Actor-Critic with Causality-Aware Entropy Regularization (ICML 2024 · similarity 0.85)
- Tackling Non-Stationarity in Reinforcement Learning via Causal-Origin Representation (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.