Research map / Reinforcement learning / Visual and representation RL
Time Reversal Symmetry for Efficient Robotic Manipulations in Deep Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TR-DRL, which exploits time reversal symmetry in robot manipulation RL via dynamics-filtered trajectory reversal augmentation for fully reversible transitions and potential-based reward shaping from reversed successful trajectories for partially reversible ones.
Problem
Sample inefficiency of deep RL in robotic manipulation; existing symmetry exploitation focuses on spatial symmetries, while temporal (time-reversal) symmetry for paired tasks such as door opening/closing and peg insertion/removal is underexplored.
Approach
Introduces partial time reversal symmetry (only object state reversible) and combines a learned inverse dynamics model with a forward-dynamics feasibility filter to generate reversed transitions for augmentation, plus a potential-based reward shaping term trained on reversed successful trajectories of the paired task.
Key results
- On MT50 12 reversible task pairs, Lang-Cond TR-SAC needs 140±25 episodes to reach 100% success on average vs 169±26 for Lang-Cond SAC (ALL row).
- Across 50 MT50 environments, Lang-Cond TR-SAC averages 196±28 episodes vs 216±28 for Lang-Cond SAC.
- Method improves sample efficiency and final success when combined with single-task SAC, task-conditioned SAC and multi-headed SAC on 10 Robosuite environments (5 pairs).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add TR-DRL to language-conditioned SAC | Lang-Cond SAC · MT50 reversible task pairs (ALL) | training episodes to 100% success rate: 169±26 -> 140±25 | Helped |
| Add TR-DRL to language-conditioned SAC | Lang-Cond SAC · MT50 all 50 environments (ALL) | training episodes to 100% success rate: 216±28 -> 196±28 | Helped |
| Add trajectory reversal augmentation on fully reversible transitions (ablation) | Single-Task SAC · Robosuite fully reversible tasks | success rate | Helped |
| Add unfiltered reversed transitions generated by the inverse dynamics model (ablation) | Single-Task SAC · Robosuite door opening/closing inward | success rate | Hurt |
| Add dynamics-aware filtering to remove invalid reversed transitions (ablation) | Trajectory reversal augmentation · Robosuite partially reversible door tasks | success rate | Helped |
| Use two separate potential models trained on own-task and reversed-task successful trajectories (ablation) | Time-reversal reward shaping · Robosuite door opening/closing inward | success rate | Helped |
| Train one potential model using only own-task successful trajectories (ablation) | Time-reversal reward shaping · Robosuite door opening/closing inward | success rate | Mixed |
| Train one potential model using only reversible-task trajectories (ablation) | Time-reversal reward shaping · Robosuite door opening/closing inward | success rate | Mixed |
| Train one joint potential model on both tasks’ successful trajectories (ablation) | Time-reversal reward shaping · Robosuite door opening/closing inward | success rate | Mixed |
| Use linear potential-value labels along successful trajectories (ablation) | Time-reversal reward shaping · Robosuite tasks | success rate | Helped |
| Vary the filtering error tolerance β among 0.01, 0.001, and 0.0001 (ablation) | Dynamics-aware filtering · Robosuite six environments | success rate | Mixed |
Surprising findings
- Naive reversal augmentation without dynamics filtering degrades performance on partially reversible door opening/closing inward tasks.
Limitations
- No real-robot experiments; all evaluation in simulation.
- Relies on prior knowledge of task structure (time reversal symmetry must be known to hold for the task pair).
- Requires a pair of related reversible tasks to be identified manually.
Experimental setup
Datasets: Robosuite environments Meta-World MT50 environments
Benchmarks: Robosuite Meta-World MT50
Metrics: success rate inter-quartile mean (IQM) of success rate training episodes to 100% success
Baselines: SAC multi-task SAC (task-conditioned) multi-headed SAC language-conditioned SAC MOORE
Builds on: Soft Actor-Critic potential-based reward shaping multi-task SAC MOORE
Closest papers by meaning
- SiT: Symmetry-invariant Transformers for Generalisation in Reinforcement Learning (ICML 2024 · similarity 0.87)
- Leveraging Partial Symmetry for Multi-Agent Reinforcement Learning (AAAI 2024 · similarity 0.86)
- Latent Mixture of Symmetries for Sample-Efficient Dynamic Learning (NeurIPS 2025 · similarity 0.86)
- Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes (AAAI 2025 · similarity 0.86)
- SkiP: When to Skip and When to Refine for Efficient Robot Manipulation (NeurIPS 2026 · similarity 0.86)
- Contractive Dynamical Imitation Policies for Efficient Out-of-Sample Recovery (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.