Research map / Reinforcement learning / Policy gradient and actor-critic
Physics-Regulated Deep Reinforcement Learning: Invariant Embeddings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Phy-DRL, a physics-regulated DRL framework combining a residual linear model-based action policy, a CLF-like safety-embedded reward derived from an LMI-based safety envelope, and physics-model-guided NN editing (link and activation masking) for critic and actor networks, with a provable safety guarantee shown on a cart-pole and a quadruped robot.
Problem
Safe deep reinforcement learning for safety-critical autonomous systems, where DNN policies are hard to verify and purely data-driven networks may violate physics laws; existing residual DRL frameworks focus on stability rather than safety and lack systematic ways to build safety-embedded rewards an
Approach
Embeds invariant prior knowledge at three levels: a residual action policy whose model-based part is a linear feedback F from known (A,B) matrices; a reward built from a Lyapunov-type quadratic term r(s(k),s(k+1)) using P from LMIs so that a safety envelope is invariant; and critic/actor networks whose Taylor-monomial inputs are augmented and whose weights are masked (link and activation editing)
Key results
- On the quadruped robot, Phy-DRL trained for 10^6 steps keeps the robot within the safety set across four test environments, while the purely data-driven DRL trained for 10^7 steps violates safety requirements in all of them.
- On the cart-pole, the sole model-based linear policy and purely data-driven DRL both yield much smaller safe areas than Phy-DRL, which renders the safety envelope invariant.
- Phy-DRL uses remarkably fewer learning parameters than the baseline networks (e.g., the PKN-15 model vs the FC MLP), per the authors' claims.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Phy-DRL instead of purely data-driven DRL with a CLF reward | Cart-pole control · Cart-pole | safety-envelope invariance | Helped |
| use Phy-DRL instead of the sole model-based policy | Cart-pole control · Cart-pole | safety area | Helped |
| use Phy-DRL instead of purely data-driven DRL | Quadruped robot control · Quadruped robot, four testing environments | safety requirement satisfaction | Helped |
| use purely data-driven DRL instead of Phy-DRL | Quadruped robot control · Quadruped robot, four testing environments | safety requirement satisfaction | Hurt |
| use the Linear or PD policy instead of Phy-DRL | Quadruped robot control · Quadruped robot, four testing environments | safety-envelope constraint satisfaction | Hurt |
| use Phy-DRL instead of the compared policies | Quadruped robot control · Quadruped robot, four testing environments | velocity-regulation performance | Helped |
Limitations
- If the linear model mismatch is large, no safe policy may exist to render the safety envelope invariant.
- The safety guarantee condition is not yet practical to test, since it requires full coverage testing within the safety envelope domain.
Experimental setup
Benchmarks: OpenAI Gym cart-pole quadruped robot simulation
Metrics: safe sample area (internal and external envelope) safety constraint satisfaction velocity regulation performance number of training steps
Baselines: purely data-driven DRL with CLF reward PD controller linear model-based policy
Builds on: actor-critic control Lyapunov function reward physics-informed neural networks physics-guided neural networks
Closest papers by meaning
- Real-DRL: Teach and Learn in Reality (NeurIPS 2025 · similarity 0.91)
- Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes (AAAI 2025 · similarity 0.88)
- A Physics-Informed Machine Learning Framework for Safe and Optimal Control of Autonomous Systems (ICML 2025 · similarity 0.86)
- Feasibility Consistent Representation Learning for Safe Reinforcement Learning (ICML 2024 · similarity 0.86)
- Verified Safe Reinforcement Learning for Neural Network Dynamic Models (NeurIPS 2024 · similarity 0.85)
- Robustness Verification of Deep Reinforcement Learning Based Control Systems Using Reward Martingales (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.