Research map / Reinforcement learning / Visual and representation RL

Constrained Meta-Reinforcement Learning for Adaptable Safety Guarantee with Differentiable Convex Programming

AAAI 2024 · Reinforcement learning › Visual and representation RL · approach: Task-inference meta-RL · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Meta-CPO, a constrained meta-RL method that differentiates through successive CPO local updates using differentiable convex optimization (cvxpylayers) to enable safe fast adaptation in non-stationary Safety Gym tasks.

Problem

Safe reinforcement learning in non-stationary environments requires fast adaptation to new tasks while satisfying cost constraints, but differentiating meta-learning losses through constrained inner-loop policy updates is intractable.

Approach

Combines MAML-style meta-learning with CPO as the inner and outer update rule, and uses differentiable convex optimization layers to backpropagate through the constrained (convexified) local policy updates, with a theoretical monotonic-improvement analysis; Euclidean metric replaces the KL Hessian for compatibility with cvxpylayers.

Key results

• Meta-CPO adapts to unseen tasks and satisfies cost limits in Car-Circle-Hazard while baselines behave unstably or fail, including when the cost bound is tightened from 10 to 5 at test time.
• TRPO converges to high returns but shows significant constraint violations; CPO is notably unstable during testing.
• Meta-CPO outperforms non-meta algorithms in both meta-training and meta-testing phases (3 random seeds).

Measured effects reported in the paper

Surprising findings

• Euclidean metric replaces the KL-divergence Hessian metric due to cvxpylayers limitations, which may compromise the monotonic improvement guarantee.

Limitations

• cvxpylayers DCO struggles with large parameter counts, restricting the approach to small policy networks and unstable in high-dimensional tasks.
• Euclidean metric substitution for the KL Hessian may break the theoretical monotonic improvement guarantee; multiple local steps are used as a mitigation.
• Evaluated only on three small Safety Gym environments with few seeds.

Experimental setup

Datasets: Safety Gym Point-Circle Safety Gym Car-Circle-Hazard Safety Gym Point-Button (procedurally generated task distributions)

Benchmarks: Point-Circle Car-Circle-Hazard Point-Button

Metrics: return cost cost constraint satisfaction

Baselines: Meta-TRPO CPO TRPO

Builds on: MAML CPO TRPO DCO (cvxpylayers)

Closest papers by meaning

• Efficient Safe Meta-Reinforcement Learning: Provable Near-Optimality and Anytime Safety (NeurIPS 2025 · similarity 0.92)
• Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning (ICLR 2025 · similarity 0.88)
• Meta-Reinforcement Learning with Universal Policy Adaptation: Provable Near-Optimality under All-task Optimum Comparator (NeurIPS 2024 · similarity 0.88)
• Absolute State-wise Constrained Policy Optimization: High-Probability State-wise Constraints Satisfaction (NeurIPS 2026 · similarity 0.87)
• Can Learned Optimization Make Reinforcement Learning Less Difficult? (NeurIPS 2024 · similarity 0.87)
• Handling Long and Richly Constrained Tasks through Constrained Hierarchical Reinforcement Learning (AAAI 2024 · similarity 0.87)