Research map / Reinforcement learning / Visual and representation RL
Make-An-Agent: A Generalizable Policy Network Generator with Behavior-Prompted Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Make-An-Agent generates MLP policy parameters with a latent conditional diffusion model conditioned on contrastive behavior embeddings from a few trajectories, enabling one-shot policy generation without fine-tuning.
Problem
Generating high-dimensional control policy parameters for new tasks or embodiments from only a few sub-optimal behavior trajectories, without downstream fine-tuning or gradient adaptation.
Approach
Treat policy generation as conditional latent diffusion over policy network parameters: an autoencoder compresses per-layer MLP weights into latents, a contrastive behavior embedding (trajectory prefix vs. post-success states) conditions a 1D conv UNet diffusion model, and the generated latents are decoded into deployable policies.
Key results
- On 8 unseen MetaWorld tasks, generated policies without fine-tuning reach Top-5 success of about 1.0 on several tasks (e.g., drawer close, faucet open, door lock) with 50-100 test trajectories.
- Qualified rate (100% success) on seen MetaWorld tasks averages above 30%, e.g., 0.45 on plate slide and 0.42 on drawer open.
- Parameter cosine similarity between generated and RL policies on unseen tasks is mostly below 0.2, suggesting the generator does not memorize training data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the contrastive behavior embeddings instead of a trajectory-encoding embedding layer in the diffusion model (ablation) | Make-An-Agent · MetaWorld unseen tasks | policy generation performance | Helped |
| Use either behavior embedding h or v alone instead of concatenating h and v (ablation) | Make-An-Agent · MetaWorld | policy generation performance | No clear effect |
| Use trajectories that are too short for behavior embeddings (ablation) | Make-An-Agent · MetaWorld | policy generation performance | Hurt |
| Increase behavior-embedding trajectory length beyond 40 steps (ablation) | Make-An-Agent · MetaWorld | policy generation performance | No clear effect |
| Use a policy network smaller than hidden size 128 (ablation) | Make-An-Agent · MetaWorld | policy performance | Hurt |
| Use a policy network larger than hidden size 128 (ablation) | Make-An-Agent · MetaWorld | policy performance | Hurt |
| Train with 1000 or fewer policy checkpoints per task (ablation) | Make-An-Agent · MetaWorld | policy generation performance | Hurt |
| Use more than 1000 policy checkpoints per task for training (ablation) | Make-An-Agent · MetaWorld | policy generation performance | No clear effect |
| Increase latent parameter representation size (ablation) | Make-An-Agent · MetaWorld | policy generation performance | Hurt |
| Reduce latent parameter representation size too far (ablation) | Make-An-Agent · MetaWorld | policy generation performance | Hurt |
Surprising findings
- Generated policies on unseen tasks are robust to Gaussian action noise in the conditioning trajectories, while baselines degrade significantly.
- Generated policy parameters differ strongly from the RL policies used to produce the test trajectories (cosine similarity mostly below 0.2 on unseen tasks).
Limitations
- Larger and more diverse policy networks were not explored due to parameter count.
- Generation capability is bounded by the quality of the parameter autoencoder.
- Unseen-task generated policies have low qualification rates (e.g., 4-16% on several unseen tasks), so success depends on sampling many candidates.
Experimental setup
Datasets: MetaWorld Robosuite walk-these-ways IsaacGym checkpoints
Benchmarks: MetaWorld Robosuite real-world Unitree Go2 locomotion
Metrics: success rate best generated policy performance average top-5 performance qualified rate cosine similarity of parameters
Baselines: Multi-task BC CARE Meta-RL with hypernetworks Decision Transformer G.pt p-diff
Builds on: DDPM hypernetworks contrastive learning (InfoNCE) walk-these-ways SAC
Closest papers by meaning
- ReGen: Generative Robot Simulation via Inverse Design (ICLR 2025 · similarity 0.89)
- One-Step Diffusion Policy: Fast Visuomotor Policies via Diffusion Distillation (ICML 2025 · similarity 0.88)
- Text-Aware Diffusion for Policy Learning (NeurIPS 2024 · similarity 0.88)
- Conditioning Matters: Training Diffusion Policies is Faster Than You Think (NeurIPS 2025 · similarity 0.88)
- Foundation Policies with Hilbert Representations (ICML 2024 · similarity 0.87)
- Diffusion Policy Policy Optimization (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.