Research map / Reinforcement learning / Policy gradient and actor-critic
LOCU: Löwdin-Orthogonalized Constraint Updates for Multi-Constraint Policy Optimization
Joonyoung Lim, Younghwan Yoo
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LOCU, which decouples interacting constraint gradients in CMDP policy optimization via symmetric Löwdin orthogonalization to obtain better-conditioned constraint update directions.
Problem
Multi-constraint policy optimization in constrained Markov decision processes, where interactions among constraint gradients produce ill-conditioned, numerically fragile update directions with cancellation or redundant overlap between constraint corrections.
Approach
Applies symmetric Löwdin orthogonalization to the constraint gradients so that constraint interactions are decoupled before forming the policy update, yielding well-conditioned constraint corrections.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply Löwdin orthogonalization with screening and Pareto recovery instead of QP-based gradient integration (SDAC actor) on goal-navigation task with heterogeneous constraints | SDAC(A) baseline · goal-navigation with Cassie-style stability constraints | return and constraint satisfaction | Improved |
| use LOCU in multi-constraint narrow-feasible-region and strongly coupled settings versus baselines | multi-constraint policy optimization · Bullet-Safety-Gym Circle v3 tightening/redundancy | return drop and constraint violations: return drop around 21% vs 26-49% and 13-32% in baselines | Improved |
| use LOCU's orthogonal basis in a reduced QP solved with Clarabel instead of the full-dimensional QP (ablation) | full QP with Clarabel · goal-navigation task | return and constraint satisfaction | Improved |
| replace LOCU's active-set solver with Clarabel interior-point solver on the full QP (ablation) | full QP baseline · goal-navigation task | recovery performance: solver flagged 2 of approximately 24,900 updates as inaccurate | Mixed |
| add gradient shaping or plain Lagrangian multipliers (PPOL variants) in the multi-constraint setting | PPO Lagrangian · Circle tasks v2 (BC, CC, DC) | circular tracking and constraint satisfaction | Hurt |
| use PCRPO soft-switching primal method in coupled multi-constraint locomotion | PCRPO · Cassie and Laikago with three constraints | convergence | Hurt |
| use LOCU versus SDAC(A) on the original locomotion tasks | SDAC(A) · Cassie and Laikago, three constraints | return with constraint satisfaction | No clear effect |
Closest papers by meaning
- Augmented Lagrangian Method for Last-Iterate Convergence for Constrained MDPs (NeurIPS 2026 · similarity 0.86)
- Learning General Parameterized Policies for Infinite Horizon Average Reward Constrained MDPs via Primal-Dual Policy Gradient Algorithm (NeurIPS 2024 · similarity 0.85)
- Sample-Efficient Constrained Reinforcement Learning with General Parameterization (NeurIPS 2024 · similarity 0.84)
- Global Convergence for Average Reward Constrained MDPs with Primal-Dual Actor Critic Algorithm (NeurIPS 2025 · similarity 0.84)
- Confident Natural Policy Gradient for Local Planning in $q_\pi$-realizable Constrained MDPs (NeurIPS 2024 · similarity 0.84)
- Policy Optimization for CMDPs with Bandit Feedback: Learning Stochastic and Adversarial Constraints (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
