Research map / Reinforcement learning / Visual and representation RL
MENTOR: Mixture-of-Experts Network with Task-Oriented Perturbation for Visual Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MENTOR, a model-free visual RL agent that replaces the MLP backbone with a mixture-of-experts and perturbs weights using top-performing agents from RL history, outperforming DrM and other visual RL baselines in simulation and on three real-world manipulation tasks.
Problem
Visual deep RL for robot manipulation suffers from low sample efficiency, and gradient conflicts in MLP backbones plus random weight perturbations limit learning in challenging tasks.
Approach
Replace the MLP policy backbone with a sparse mixture-of-experts so gradients route to specialized experts, and replace random perturbation candidates with weights sampled from a Gaussian fit to a buffer of top-performing agents' weights from training history.
Key results
- Real-world: MENTOR achieves an average 83% success rate across three real-world tasks vs. 32% for the strongest model-free baseline (DrM).
- Simulation: about 17% and 10% higher episode reward than the top baseline on Dog Stand and Dog Walk; reduces training frames by 70% on Hammer (Sparse) vs. best baseline.
- Ablation on real-world Peg Insertion/Cable Routing/Tabletop Golf: removing MoE drops overall performance by nearly 30%; further replacing task-oriented perturbation with random perturbation drops another ~30%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add task-oriented perturbation instead of random perturbation (ablation) | MENTOR · Hopper Hop, Disassemble, Coffee Push, and Hammer | normalized sample efficiency (training time): Hopper Hop: 1 -> 0.6167; Disassemble: 0.8505 -> 0.7056; Coffee Push: 0.9481 -> 0.8066; Hammer: 0.875 -> 0.7167 | Helped |
| add MoE instead of using an MLP backbone (ablation) | MENTOR · Hopper Hop, Disassemble, Coffee Push, and Hammer | normalized sample efficiency (training time): Hopper Hop: 0.85 -> 0.6167; Disassemble: 1 -> 0.7056; Coffee Push: 1 -> 0.8066; Hammer: 1 -> 0.7167 | Helped |
| add a pretrained visual encoder instead of training a CNN from scratch (ablation) | MENTOR · Real-world Peg Insertion, Cable Routing, and Tabletop Golf | performance: 9% | Helped |
| add MoE instead of using an MLP backbone (ablation) | MENTOR · Peg Insertion, Cable Routing, and Tabletop Golf | success rate: Triangle: 0.7 -> 1.0; Arrow: 0.6 -> 1.0; Cable Routing: 0.45 -> 0.8; Tabletop Golf: 0.55 -> 0.7 | Helped |
| use task-oriented perturbation rather than basic random perturbation (ablation) | MENTOR · Simulation tasks | sample efficiency | Helped |
| increase the number of MoE experts from 4 to 8 or 32 while keeping top_k = 4 (ablation) | MENTOR · Meta-World Hammer (Sparse) | performance | No clear effect |
| replace MENTOR with DrM | Real-world MENTOR · Peg Insertion, Cable Routing, and Tabletop Golf | success rate: Star: 1.0 -> 0.5; Triangle: 1.0 -> 0.2; Arrow: 1.0 -> 0.1; Cable Routing: 0.8 -> 0.2; Tabletop Golf: 0.7 -> 0.5 | Mixed |
Surprising findings
- MENTOR has lower FPS than DrM and DrQ-v2 in simulation (37 vs 55 and 78 on Hopper), attributed to a plain MoE implementation that runs all experts.
- Pretrained visual encoder gave about 9% real-world gain but no significant gain in simulation.
Limitations
- Evaluated only on single tasks with a single robot embodiment.
- Plain MoE implementation lowers wall-clock time efficiency compared with baselines.
- Real-world reward partly relies on approximate estimation (e.g., camera-based ball localization) and trained visual classifiers.
Experimental setup
Datasets: DeepMind Control Suite Meta-World Adroit
Benchmarks: DeepMind Control Suite Meta-World Adroit Peg Insertion (real-world) Cable Routing (real-world) Tabletop Golf (real-world)
Metrics: episode reward success rate training frames to threshold FPS
Baselines: DrM DrQ-v2 ALIX TACO
Builds on: DrQ-v2 DrM dormant neurons mixture-of-experts load balancing
Closest papers by meaning
- Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes (AAAI 2025 · similarity 0.89)
- RL-CycleGAN: Reinforcement Learning Aware Simulation-to-Real (CVPR 2020 · similarity 0.88)
- Robot-R1: Reinforcement Learning for Enhanced Embodied Reasoning in Robotics (NeurIPS 2025 · similarity 0.87)
- PriorVLA: Prior-Preserving Adaptation for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.86)
- GoalLadder: Incremental Goal Discovery with Vision-Language Models (NeurIPS 2025 · similarity 0.86)
- Mixture-of-Experts Meets In-Context Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.