Research map / Reinforcement learning / Visual and representation RL
MeMo: Meaningful, Modular Controllers via Noise Injection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MeMo, which pretrains modular robot controllers (boss plus per-assembly worker modules) with noise injection during imitation learning so that the modules can be reused for faster RL on new morphologies and tasks.
Problem
Robots built from standardized assemblies must be retrained from scratch when the morphology or task changes; existing transfer methods (GNN message passing, Transformers) need large multi-robot datasets or overfit in data-scarce single-robot settings.
Approach
A modularity objective that makes per-assembly worker modules invariant to noise in the boss signal, optimized by injecting Gaussian noise into the boss output during DAgger imitation learning, so workers take responsibility for coordination within their assembly and transfer across morphologies and tasks.
Key results
- On the 6-to-12 leg centipede and 10-leg hybrid transfers, MeMo is about 2x more sample efficient than the best baseline and converges to significantly higher reward.
- MeMo reaches the final performance of the closest baseline within half of the total timesteps across structure transfer tasks.
- Ablations show noise injection is the critical component; MeMo without noise injection, with L1/L2 regularization, or with Jacobian penalty all yield weaker transfer.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pretrain reusable modules with MeMo noise injection | MeMo · 6-leg to 12-leg centipede, Frozen Terrain | training efficiency: 2× more sample efficient than the best baseline | Helped |
| pretrain reusable modules with MeMo noise injection | MeMo · 6-leg to 10-leg hybrid, Frozen Terrain | training efficiency: 2× more sample efficient than the best baseline | Helped |
| pretrain reusable modules with MeMo noise injection | MeMo · 6-leg to 10-leg worm, Frozen Terrain | training efficiency | Helped |
| pretrain reusable modules with MeMo noise injection | MeMo · 4-finger to 5-finger claw, cube grasping | training efficiency | Helped |
| pretrain modules with MeMo and fine-tune end-to-end | MeMo · Centipede, Frozen to Ridged Terrain | training efficiency | Helped |
| pretrain modules with MeMo and fine-tune end-to-end | MeMo · Centipede, Frozen to Gap Terrain | training efficiency | Helped |
| pretrain modules with MeMo and fine-tune end-to-end | MeMo · Centipede, Frozen to Stepped Terrain | training efficiency | Helped |
| transfer pretrained modules from cube to sphere grasping | MeMo · 4-finger claw, cube to sphere | training efficiency | Helped |
| optimize the noise injection objective instead of the dual loss (ablation) | MeMo · Centipede, Worm, Hybrid, and Claw | sum of modularity objectives: -33.115 -> -33.518; -35.896 -> -39.295; -27.849 -> -30.536; 0.279 -> -8.215 | Helped |
| add Gaussian noise injection during pretraining (ablation) | MeMo modular controller · 6-leg centipede, Frozen Terrain, same-structure transfer | training efficiency | Helped |
| replace Gaussian noise injection with L1 regularization on the boss output (ablation) | MeMo modular controller · 6-leg centipede, Frozen Terrain, same-structure transfer | training efficiency | Hurt |
| replace Gaussian noise injection with L2 regularization on the boss output (ablation) | MeMo modular controller · 6-leg centipede, Frozen Terrain, same-structure transfer | training efficiency | Hurt |
Surprising findings
- Pretraining modules with noise injection in RL (naive or selective) did not improve training efficiency; only the IL-based pretraining helped.
- On the 10 leg worm, the GNN baseline NerveNet-Conv is competitive with MeMo in final performance because multi-hop communication suits the shorter legs.
Limitations
- Experiments only in simulation and only with PPO and MLP-based architectures.
- Task transfer requires end-to-end finetuning because the architecture does not encode task semantics; transfer is limited to incrementally different robots from a single robot and environment.
Experimental setup
Datasets: RoboGrammar locomotion tasks DiffRedMax grasping tasks
Benchmarks: RoboGrammar locomotion (Frozen, Ridged, Gap, Stepped terrains) DiffRedMax grasping (cube, sphere)
Metrics: average episode reward training sample efficiency (timesteps to reach final baselin sum of modularity objectives normalized Jacobian singular values
Baselines: RL (MLP) RL (Modular) NerveNet-Conv NerveNet-Snowflake MetaMorph MeMo (no NI) MeMo (L1) MeMo (L2) MeMo (Jacobian)
Builds on: PPO DAgger NerveNet MetaMorph Snowflake
Closest papers by meaning
- One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control (ICML 2020 · similarity 0.87)
- MetaMorph: Learning Universal Controllers with Transformers (ICLR 2022 · similarity 0.87)
- DittoGym: Learning to Control Soft Shape-Shifting Robots (ICLR 2024 · similarity 0.85)
- Meta-Controller: Few-Shot Imitation of Unseen Embodiments and Tasks in Continuous Control (NeurIPS 2024 · similarity 0.84)
- Distilling Morphology-Conditioned Hypernetworks for Efficient Universal Morphology Control (ICML 2024 · similarity 0.84)
- Think Small, Act Big: Primitive Prompt Learning for Lifelong Robot Manipulation (CVPR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.