Research map / Reinforcement learning / Policy gradient and actor-critic
Enhancing Off-Policy Constrained Reinforcement Learning through Adaptive Ensemble C Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Adaptive Ensemble C-learning (AEC) with a PID-adapted ensemble size K and a First C Buffer to reduce C-value estimation bias, applied to SAC and TQC for off-policy constrained RL.
Problem
Off-policy constrained RL struggles to accurately estimate the C-function (expected cumulative cost), because the Lagrange multiplier fluctuates and the replay buffer contains non-initial states, causing over- or underestimation that hurts constraint satisfaction.
Approach
Ensemble C-value targets use the mean of the highest or lowest N-K (or N+K) C estimates, with K adapted online by a PID controller on the C-value error from Monte Carlo returns; a First C Buffer stores initial states so the constraint estimate matches the episode-start threshold.
Key results
- AECSAC reaches reward 2954.377 on Walker2d-v3 versus 2186.234 for FOCOPS (best on-policy baseline) while keeping cost near threshold (77.945 vs 80).
- On AntCircle (threshold 50), AECTQC achieves reward 4021.963 versus 763.043 for CUP.
- Ablations show removing AEC causes learning-curve fluctuations and removing FCB causes constraint undershoot (Walker2d) or overshoot (Swimmer).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply adaptive ensemble C-learning and First C Buffer to SAC | AECSAC · Ant-v3 | Reward: 1469.299±59.839 -> 1645.73±125.594 | Helped |
| Apply adaptive ensemble C-learning and First C Buffer to SAC | AECSAC · HalfCheetah-v3 | Reward: 1485.953±47.143 -> 1561.42±20.705 | Helped |
| Apply adaptive ensemble C-learning and First C Buffer to SAC | AECSAC · Humanoid-v3 | Reward: 4577.634±108.064 -> 5102.084±235.089 | Helped |
| Apply adaptive ensemble C-learning and First C Buffer to SAC | AECSAC · Swimmer-v3 | Reward: 36.669±5.651 -> 45.787±3.478 | Helped |
| Apply adaptive ensemble C-learning and First C Buffer to SAC | AECSAC · Walker2d-v3 | Reward: 2186.234±316.851 -> 2954.377±54.398 | Helped |
| Remove adaptive ensemble C-learning (ablation) | AECSAC · Walker2d-v3 and Swimmer-v3 | Reward | Hurt |
| Remove adaptive ensemble C-learning (ablation) | AECTQC · Walker2d-v3 and Swimmer-v3 | Reward | Hurt |
| Replace First C Buffer with replay-buffer state-action samples for estimating constraint cost (ablation) | AECSAC · Walker2d-v3 and Swimmer-v3 | Reward | Hurt |
| Replace First C Buffer with replay-buffer state-action samples for estimating constraint cost (ablation) | AECTQC · Walker2d-v3 and Swimmer-v3 | Reward | Hurt |
| Vary the number of C-value estimates N (10, 15, or 20) (ablation) | AECSAC · Walker2d-v3 | Reward | No clear effect |
Surprising findings
- Across the tested Lagrange multiplier values, C-value bias changes sign, so no single fixed ensemble aggregation (min, mean, or max) corrects it throughout training.
Limitations
- Theoretical bias bounds assume uniform random approximation error and a state with M actions sharing identical true Q and C values.
- Evaluated only on simulated MuJoCo and Safety-Gymnasium tasks, not real robots despite the real-world motivation.
Experimental setup
Datasets: MuJoCo speed-limit tasks (Ant, HalfCheetah, Humanoid, Swimmer, Walker2d) Safety-Gymnasium circle tasks (AntCircle, HumanoidCircle) Safety-Gymnasium PointGoal1, CarGoal1
Benchmarks: MuJoCo speed-limit tasks (Ant-v3, HalfCheetah-v3, Humanoid-v3, Swimmer-v3, Walker2d-v3) AntCircle HumanoidCircle PointGoal1 CarGoal1
Metrics: episode return episode cost C-value bias
Baselines: FOCOPS CUP CPO PPOLag P3O CVPO SACLag CDMPO
Builds on: SAC TQC Lagrangian relaxation PID Lagrangian CVPO
Closest papers by meaning
- Adaptive Scaling of Policy Constraints for Offline Reinforcement Learning (ICLR 2026 · similarity 0.88)
- An Offline Adaptation Framework for Constrained Multi-Objective Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
- Last-Iterate Global Convergence of Policy Gradients for Constrained Reinforcement Learning (NeurIPS 2024 · similarity 0.87)
- Constrained Reinforcement Learning Under Model Mismatch (ICML 2024 · similarity 0.87)
- Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning (AAAI 2025 · similarity 0.87)
- Revisiting Cooperative Off-Policy Multi-Agent Reinforcement Learning (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.