Research map / Reinforcement learning / Visual and representation RL
PEAC: Unsupervised Pre-training for Cross-Embodiment Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes cross-embodiment unsupervised RL (CEURL), formalized as a Controlled Embodiment MDP, and PEAC, a pre-training method with an embodiment-discriminator intrinsic reward that can be combined with LBS and DIAYN.
Problem
Cross-embodiment RL agents trained on specific tasks learn knowledge coupled to those tasks rather than to embodiments, so they generalize poorly to new embodiments and fine-tune slowly; the paper asks how to pre-train embodiment-aware, task-agnostic knowledge without reward.
Approach
Formulates pre-training across a distribution of embodiments as a CE-MDP, derives that the worst-case pre-training objective reduces to a KL term equal to an embodiment-posterior log-likelihood, and turns it into an intrinsic reward (log prior minus log discriminator posterior) that rewards visiting states where the embodiment is hard to identify; PEAC can be added to existing exploration or skill
Key results
- In state-based DMC generalization to unseen embodiments, PEAC reaches 631.0 on Quadruped-mass and 573.7 on Quadruped-damping versus the best baseline BeCL at 475.6 and 421.9 (Table 1 left).
- In image-based DMC, PEAC-LBS attains the best return on all three embodiment settings (754.8, 740.8, 742.1), above LBS (658.2, 730.7, 732.7) and Choreographer (687.8, 682.3, 724.6).
- PEAC-LBS is the best method from 1M pre-training steps onward in image-based DMC ablations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PEAC instead of BeCL | State-based DMC · Walker-mass, unseen embodiments | average return: 544.5 $\pm$ 258.7 -> 491.3 $\pm$ 250.1 | Hurt |
| Use PEAC instead of CIC | State-based DMC · Quadruped-mass, unseen embodiments | average return: 602.2 $\pm$ 193.8 -> 631.0 $\pm$ 235.7 | Helped |
| Use PEAC instead of DIAYN | State-based DMC · Quadruped-damping, unseen embodiments | average return: 397.0 $\pm$ 159.9 -> 573.7 $\pm$ 220.3 | Helped |
| Combine PEAC with DIAYN | Image-based DMC · Walker-mass, unseen embodiments | average return: 609.4 $\pm$ 260.2 -> 621.9 $\pm$ 235.1 | Helped |
| Combine PEAC with DIAYN | Image-based DMC · Quadruped-mass, unseen embodiments | average return: 566.9 $\pm$ 158.0 -> 556.1 $\pm$ 179.4 | Hurt |
| Combine PEAC with DIAYN | Image-based DMC · Quadruped-damping, unseen embodiments | average return: 558.6 $\pm$ 169.5 -> 557.2 $\pm$ 160.7 | No clear effect |
| Combine PEAC with LBS | Image-based DMC · Walker-mass, unseen embodiments | average return: 687.8 $\pm$ 222.7 -> 754.8 $\pm$ 214.6 | Helped |
| Combine PEAC with LBS | Image-based DMC · Quadruped-mass, unseen embodiments | average return: 730.7 $\pm$ 162.3 -> 740.8 $\pm$ 171.3 | Helped |
| Combine PEAC with LBS | Image-based DMC · Quadruped-damping, unseen embodiments | average return: 732.7 $\pm$ 142.5 -> 742.1 $\pm$ 165.2 | Helped |
| Increase cross-embodiment pre-training timesteps (ablation) | PEAC-LBS and PEAC-DIAYN · Image-based DMC downstream tasks | fine-tuned performance | Helped |
Surprising findings
- The paper's worked example suggests vertices of the embodiment-averaged state-distribution set need not be deterministic policies, which challenges the single-embodiment assumption.
- Pure skill-discovery methods like DIAYN fall further behind exploration methods in the cross-embodiment image setting than in the single-embodiment setting.
Limitations
- Assumes embodiments share similar structure, so it may struggle with extremely different embodiments.
- Existing unsupervised RL methods still struggle on more challenging downstream tasks.
Experimental setup
Datasets: DeepMind Control Suite (state and image) URLB downstream tasks Robosuite (Panda, IIWA, Kinova3, Jaco) Isaacgym Unitree A1 (12 joint-failure embodiments)
Benchmarks: state-based DMC image-based DMC Robosuite Isaacgym A1-disabled real-world Aliengo legged locomotion
Metrics: expert normalized score mean median interquartile mean optimality gap average return
Baselines: ICM RND Disagreement ProtoRL LBS DIAYN SMM APS CIC BeCL Plan2Explore APT
Builds on: URLB DIAYN LBS DreamerV2 DrQ
Closest papers by meaning
- Foundation Policies with Hilbert Representations (ICML 2024 · similarity 0.88)
- Dual RL: Unification and New Methods for Reinforcement and Imitation Learning (ICLR 2024 · similarity 0.88)
- CrossLoco: Human Motion Driven Control of Legged Robots via Guided Unsupervised Reinforcement Learning (ICLR 2024 · similarity 0.88)
- Unifying Goal-Conditioned RL and Unsupervised Skill Learning via Control-Maximization (NeurIPS 2026 · similarity 0.87)
- Behavioral Exploration: Learning to Explore via In-Context Adaptation (ICML 2025 · similarity 0.87)
- METRA: Scalable Unsupervised RL with Metric-Aware Abstraction (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.