Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Real-DRL: Teach and Learn in Reality
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Real-DRL, a runtime-learning framework where a DRL student learns from its own experience and from a physics-model-based PHY-Teacher that takes over via a trigger when the plant nears safety boundaries, with safety-informed batch sampling.
Problem
Safe and high-performance DRL in real physical plants under unknown unknowns and the Sim2Real gap, where simulator-trained policies can be unsafe and existing fault-tolerant or safe DRL methods rely on assumed-good backup controllers.
Approach
A DRL-Student learns via dual replay buffers (self-learning and teaching-to-learn from a PHY-Teacher), with batch composition weighted by a Lyapunov-like safety-status indicator V(s); PHY-Teacher computes LMI-based real-time patch controllers that return the state to the self-learning space, and a Trigger switches control between them.
Key results
- In Isaac Gym navigation, Real-DRL reached success with no falls or collisions, travel time 45.34 s and total energy 21742 J, versus 55.53 s and 26818 J for PHY-Teacher alone.
- On the real quadruped, Real-DRL kept the robot safe where the simulator-trained Phy-DRL policy did not, and DRL-Student learned to be safe within about 20 episodes (300 s) of PHY-Teacher guidance.
- Ablations on cart-pole show teaching-to-learn improves sample efficiency and safety-informed batch sampling improves safety in corner-case environments.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add safety-informed batch sampling (SBS) instead of using a single replay buffer without SBS (ablation) | Real-DRL DRL-Student · Cart-pole imbalanced corner-case environment | safety assurance | Helped |
| Add the teaching-to-learn mechanism (ablation) | Real-DRL · Cart-pole | episode-average reward | Helped |
| Enable automatic hierarchy learning with PHY-Teacher deactivated in episodes 5 and 50 (ablation) | Real-DRL · Cart-pole | PHY-Teacher activation ratio | Helped |
| Add PHY-Teacher support during runtime learning (ablation) | Real-DRL · Real quadruped robot indoor environment | episodes to learn safe behavior: 20 episodes, i.e., 300 seconds | Helped |
| Use Real-DRL instead of PHY-Teacher alone | Quadruped robot navigation in NVIDIA Isaac Gym wild environment · NVIDIA Isaac Gym wild environment | Travel Time (s): 55.5327 -> 45.3383 | Helped |
| Use Real-DRL instead of PHY-Teacher alone | Quadruped robot navigation in NVIDIA Isaac Gym wild environment · NVIDIA Isaac Gym wild environment | Total Energy (J): 26817.68 -> 21742.42 | Helped |
Surprising findings
- The simulator-trained Phy-DRL policy deployed directly to the real robot could not guarantee safety, and continual fine-tuning also was not sufficient in the reported comparisons.
Limitations
- Trigger must continuously monitor safety status, which imposes monitoring burden (authors propose reachability-based early warning as future work).
- Safety guarantees rely on a model-mismatch bound assumption that may be violated in unpredictable environments; an adaptive mechanism is only a partial remedy.
Experimental setup
Datasets: self-learning experience from real quadruped teaching-to-learn experience from PHY-Teacher NVIDIA Isaac Gym quadruped simulation cart-pole simulation
Benchmarks: real quadruped robot indoor experiment quadruped in NVIDIA Isaac Gym wild environment cart-pole ablation
Metrics: is fall collision success travel time (s) average power (W) total energy (J) episode return CoM height and velocity
Baselines: CLF-DRL Phy-DRL Neural Simplex Runtime Assurance Continual Phy-DRL
Builds on: Simplex architecture Phy-DRL DDPG soft actor-critic
Closest papers by meaning
- Physics-Regulated Deep Reinforcement Learning: Invariant Embeddings (ICLR 2024 · similarity 0.91)
- Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes (AAAI 2025 · similarity 0.87)
- Don’t Trade Off Safety: Diffusion Regularization for Constrained Offline RL (NeurIPS 2025 · similarity 0.86)
- HMARL-CBF – Hierarchical Multi-Agent Reinforcement Learning with Control Barrier Functions for Safety-Critical Autonomous Systems (NeurIPS 2025 · similarity 0.86)
- SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Constrained Learning (NeurIPS 2025 · similarity 0.86)
- A Provable Approach for End-to-End Safe Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.