Research map / Multi-agent systems and games / Cooperative multi-agent RL
HMARL-CBF – Hierarchical Multi-Agent Reinforcement Learning with Control Barrier Functions for Safety-Critical Autonomous Systems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HMARL-CBF, a two-level skill-based multi-agent RL method where a high-level joint policy selects skills and a low-level CBF-based QP policy executes them safely, evaluated in MetaDrive and lidar environments.
Problem
Safe cooperative policy learning for many agents in safety-critical, partially observable settings where pointwise-in-time safety constraints must hold, which flat MARL and CMDP-based methods do not provide.
Approach
Decompose the problem into a high-level centralized-training joint skill policy (MAPPO/QMIX-style) and a low-level per-agent skill policy whose actions are produced by a parameterized CBF-constrained QP, so forward invariance of the CBF constraints guarantees safety during training and deployment; the high-level task advantage is also fed into low-level reward.
Key results
- On MetaDrive merging, HMARL-CBF (on-policy) reaches 0.99 success vs 0.43 for IPPO and 0.48 for CoPo.
- On bottleneck, HMARL-CBF reaches 0.99 success with 241 time units vs 0.75 success and 651 time for CoPo.
- Flat-policy MAPPO variants achieve 0 success in all environments, per the ablation table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use HMARL-CBF instead of the strongest reported baseline | MetaDrive multi-agent traffic scenarios · Merging | success rate: 0.48_{\pm0.01} -> 0.99_{\pm0.01} | Helped |
| use HMARL-CBF instead of the strongest reported baseline | MetaDrive multi-agent traffic scenarios · Intersection | success rate: 0.73_{\pm0.09} -> 0.97_{\pm0.04} | Helped |
| use HMARL-CBF instead of the strongest reported baseline | MetaDrive multi-agent traffic scenarios · Roundabout | success rate: 0.85_{\pm0.05} -> 0.99_{\pm0.01} | Helped |
| use HMARL-CBF instead of the strongest reported baseline | MetaDrive multi-agent traffic scenarios · Bottleneck | success rate: 0.75_{\pm0.07} -> 0.99_{\pm0.00} | Helped |
| use HMARL-CBF instead of the strongest reported baseline | MetaDrive multi-agent traffic scenarios · Tollgate | success rate: 0.11_{\pm0.04} -> 0.99_{\pm0.02} | Helped |
| remove the hierarchy and use a flat MAPPO policy (ablation) | HMARL-CBF · MetaDrive environments | success rate: 0.99_{\pm0.01} -> 0 | Hurt |
| replace cooperative high-level policy with uniform random skill selection (ablation) | HMARL-CBF · MetaDrive environments | success rate: 0.99_{\pm0.01} -> 0.94_{\pm0.09} | Hurt |
| randomly drop road-boundary constraints from the low-level policy (ablation) | HMARL-CBF · MetaDrive environments | success rate: 0.99_{\pm0.01} -> 0.91_{\pm0.04} | Hurt |
| replace the CBF skill policy with a parameterized LQR policy (ablation) | HMARL-CBF · Intersection | success rate: 0.97_{\pm0.04} -> 0.14_{\pm0.02} | Hurt |
| fix low-level policy parameters at feasible-space boundaries (ablation) | HMARL-CBF · Merging | success rate: 0.99_{\pm0.01} -> 0.52_{\pm0.06} | Hurt |
| transfer low-level skills learned in Intersection to Merging | HMARL-CBF · Merging | success rate: 0.99_{\pm0.01} -> 0.96_{\pm0.016} | Hurt |
| transfer low-level skills learned in Intersection to Roundabout | HMARL-CBF · Roundabout | success rate: 0.99_{\pm0.01} -> 0.96_{\pm0.016} | Hurt |
Surprising findings
- A flat MAPPO policy with safety penalties fails to learn at all (0 success) in every environment.
- Increasing safety penalty factors for baselines does not improve their success rates in the bottleneck environment.
Limitations
- Safety guarantees rely on CBF constraints with a dynamic/kinematic bicycle model; the environment uses pybullet physics, so model mismatch is possible (robust CBFs only mentioned as future extension).
- Performance drops with narrow lanes in the intersection environment, and success relies on curriculum learning for roundabout and intersection.
- Some baselines are reported as having 2000+ hours of driving data, making comparison sample-complexity claims unclear.
Experimental setup
Datasets: MetaDrive simulator environments (merging, intersection, roundabout, bottleneck, tollgate) lidar-based Target, Spread, Target-Bicycle environments
Benchmarks: MetaDrive merging MetaDrive intersection MetaDrive roundabout MetaDrive bottleneck MetaDrive tollgate lidar Target lidar Spread lidar Target-Bicycle
Metrics: success rate success-weighted average time success-weighted average energy reward success/safety rate
Baselines: IPPO CL (curriculum learning) MFPO CoPo DGPPO InforMARL MAPPO-Lagrangian
Builds on: MAPPO QMIX PPO SAC CoPo
Closest papers by meaning
- Scalable Constrained Policy Optimization for Safe Multi-agent Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- Discrete GCBF Proximal Policy Optimization for Multi-agent Safe Optimal Control (ICLR 2025 · similarity 0.87)
- Scalable Safe Policy Improvement for Factored Multi-Agent MDPs (ICML 2024 · similarity 0.87)
- Handling Long and Richly Constrained Tasks through Constrained Hierarchical Reinforcement Learning (AAAI 2024 · similarity 0.87)
- Real-DRL: Teach and Learn in Reality (NeurIPS 2025 · similarity 0.86)
- Long-Term Safe Reinforcement Learning with Binary Feedback (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.