Research map / Reinforcement learning / Visual and representation RL
A Transfer Approach Using Graph Neural Networks in Deep Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TURRET, a GNN-based policy network with attention and set-transformer readout that maps heterogeneous robots into a unified state embedding space, plus distance-weighted lateral transfer from multiple source policies for cross-domain multi-source transfer in MuJoCo.
Problem
Multi-source transfer reinforcement learning when source and target tasks have different state-action spaces (e.g., robots with different joint counts or morphologies), where existing methods assume matching spaces or fixed numbers of sources.
Approach
Uses attention-based GNN policies with a set-transformer readout to embed states from different robot morphologies into a shared space, then weights source policies per state by embedding distance and transfers them via lateral connections, with no extra optimization objectives beyond PPO.
Key results
- TURRET consistently outperforms all baselines across size transfer targets Centipede-12, -16, -20 (Figure 3 a-c).
- In morphology transfer, TURRET significantly improves learning efficiency over all baselines on Humanoid, while CAT matches TURRET only on Centipede-8.
- Ablations on Centipede-{4,6}->16 show removing attention or replacing distance weighting with average performance both reduce return relative to full TURRET.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TURRET instead of PPO (MLP) for cross-domain transfer | Centipede-4 → Centipede-16 · Centipede-4 → Centipede-16 trajectories | average Euclidean distance: 33.58 -> 20.51 | Helped |
| Use TURRET instead of PPO (MLP) for cross-domain transfer | Centipede-4 → Centipede-20 · Centipede-4 → Centipede-20 trajectories | average Euclidean distance: 54.69 -> 20.62 | Helped |
| Use TURRET instead of PPO (MLP) for cross-domain transfer | Hopper → Walker2d · Hopper → Walker2d trajectories | average Euclidean distance: 18.46 -> 16.98 | Helped |
| Use TURRET instead of PPO (MLP) for cross-domain transfer | HalfCheetah → Humanoid · HalfCheetah → Humanoid trajectories | average Euclidean distance: 21.17 -> 18.96 | Helped |
| Add attention during GNN aggregation (ablation) | TURRET · Centipede-{4,6} → Centipede-16 | episode rewards | Helped |
| Add embedding-distance similarity weighting instead of average-performance weighting (ablation) | TURRET · Centipede-{4,6} → Centipede-16 | episode rewards | Helped |
| Use TURRET instead of tested baselines | Centipede-{4,6} → Centipede-{12,16,20} · Centipede size transfer | average episodic return | Helped |
| Use TURRET instead of CAT | HalfCheetah & Ant → Centipede-8 · Centipede-8 | average episodic return | No clear effect |
| Use TURRET instead of tested baselines | Hopper & Centipede-4 → Walker2d; HalfCheetah & Ant → Humanoid · Walker2d and Humanoid morphology transfer | learning efficiency | Helped |
Limitations
- Assumes source and target tasks share a similar goal, so state similarity reflects task similarity.
- Assumes clean state composition where each feature corresponds to a robot joint; noisy or irrelevant features are not handled.
- Requires structured robot morphology information in advance.
Experimental setup
Datasets: MuJoCo Walker2d MuJoCo Ant MuJoCo Hopper MuJoCo HalfCheetah MuJoCo Humanoid MuJoCo Centipede-n
Benchmarks: MuJoCo Centipede-12/16/20 MuJoCo Walker2d MuJoCo Centipede-8 MuJoCo Humanoid
Metrics: average episodic return Euclidean distance between t-SNE projected trajectories
Baselines: PPO CAT NerveNet Snowflake NerveNet+fine-tune SWAT
Builds on: NerveNet PPO CAT Graph attention networks Set Transformer
Closest papers by meaning
- NerveNet: Learning Structured Policy with Graph Neural Networks (ICLR 2018 · similarity 0.87)
- Model-Based Transfer Learning for Contextual Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
- SF-DQN: Provable Knowledge Transfer using Successor Feature for Deep Reinforcement Learning (ICML 2024 · similarity 0.86)
- Knowledge Transfer in Multi-Task Deep Reinforcement Learning for Continuous Control (NeurIPS 2020 · similarity 0.86)
- Structure-Aware Transformer Policy for Inhomogeneous Multi-Task Reinforcement Learning (ICLR 2022 · similarity 0.86)
- Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.