Research map / Reinforcement learning / Visual and representation RL
Robot Fleet Learning via Policy Merging
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes policy merging (PoMe / Fleet-Merge) that aggregates weights of separately trained recurrent visuomotor policies by aligning permutation and similarity symmetries, enabling fleet learning without sharing data.
Problem
Robot fleets generate heterogeneous streaming data in silos that cannot be centralized or transmitted cheaply, so the problem is how to consolidate skills from separately trained policies into one policy without sharing data. Naive weight averaging fails for recurrent policies because of permutation
Approach
Define permutation symmetries of deep RNN policies (permuting layers per timestep shared across time) and connect them to similarity transformations of linear dynamical controllers, then merge policies via alternating linear-sum-assignment or gradient-based soft permutations, extended to multi-round merging during training (Fed-Rebasin).
Key results
- Fleet-Merge is reported to outperform baselines by over 50% by accounting for RNN permutation symmetries.
- On Drake tool-use tasks, permutation alignment achieves almost zero performance barrier between separately trained feedforward policies.
- On Meta-World, the mode connectivity curve is basically flat, and the merged policy nearly matches joint multi-task training on 50 tasks without sharing data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Alternation-based permutation alignment (Hungarian/linear sum assignment) for merging linear policies | linear dynamic policies (LQG setting) · linear control single-task and multi-task merging | closed-loop rollout metric / performance barrier | Mixed |
| Gradient-based joint optimization of invertible transforms instead of permutation-only alternation (ablation) | linear dynamic policy merging · linear control merging | closed-loop rollout metric | Helped |
| Naive weight averaging of linear policies instead of alignment-aware merging | linear static feedback policies · linear control merging | closed-loop rollout metric | Helped |
| Merging policies trained on different tasks using alternation-based algorithm vs naive averaging in multi-task decentralized learning | multi-task Meta-World policies under FedAvg-style communication · Meta-World 50 tasks | success rate | Helped |
| Permutation alignment of feedforward policies before weight interpolation | feedforward visuomotor policies on Drake tool-use tasks · Drake Fleet-Tools | performance barrier | Helped |
Surprising findings
- Averaging linear policies often nearly matches joint training on pooled data in the single-task setting.
Limitations
- Analysis for linear dynamic output feedback policies is marked as ongoing in the text.
- Theoretical treatment of behavior cloning in multi-task settings is left for future work.
- Permutation symmetry analysis is restricted to permutations, which may not capture all nonlinear alignments.
Experimental setup
Datasets: Meta-World 50-task benchmark ShapeNet Drake tool-use scripted expert demonstrations LQG expert trajectories
Benchmarks: Meta-World Fleet-Tools linear LQG control
Metrics: success rate task performance barrier imitation loss barrier closed-loop rollout error
Baselines: FedAvg naive weight averaging single-task training multitask joint training Git Re-Basin
Builds on: Git Re-Basin FedAvg Sinkhorn operator Elman RNN
Closest papers by meaning
- STRAP: Robot Sub-Trajectory Retrieval for Augmented Policy Learning (ICLR 2025 · similarity 0.86)
- Generalizable Domain Adaptation for Sim-and-Real Policy Co-Training (NeurIPS 2025 · similarity 0.86)
- Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformers (NeurIPS 2024 · similarity 0.85)
- Efficient Robotic Policy Learning via Latent Space Backward Planning (ICML 2025 · similarity 0.85)
- BAKU: An Efficient Transformer for Multi-Task Policy Learning (NeurIPS 2024 · similarity 0.85)
- Active Fine-Tuning of Multi-Task Policies (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.