Research map / Reinforcement learning / Policy gradient and actor-critic
Autoregressive Policy Optimization for Constrained Allocation Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PASPO, an autoregressive policy over convex-polytope allocation spaces that guarantees hard linear constraint satisfaction by sequentially sampling each allocation within LP-computed bounds, plus a de-biasing initialization for the beta distributions.
Problem
Constrained allocation tasks require strictly satisfying linear hard constraints on resource allocations at every step, but existing safe RL methods only handle soft constraints and hard-constraint methods rely on post-hoc action correction or are limited in constraint types.
Approach
Decompose the polytope action space into sequential sub-problems: for each entity, solve LPs for the feasible interval given prior allocations, sample from a normalized four-parameter beta distribution over that interval, and update the remaining polytope; add a de-biasing term to the beta initialization so the initial sampling approximates uniform over the joint action space.
Key results
- PASPO converges faster to higher average return than CPO, CUP, IPO, P3O, PPO-Lagrangian and OptLayer across all three environments.
- Only PASPO and OptLayer guarantee zero constraint violations; soft-constraint baselines show violations during training.
- Ablation on the synthetic benchmark shows that without de-biased initialization learning is slower and converges prematurely to a sub-optimal policy; with de-biasing the allocation order has no significant effect.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PASPO instead of the compared baselines | PPO on PASPO · Synthetic benchmark, Portfolio Optimization, Compute Load Distribution | average episode reward | Helped |
| Generate actions with PASPO rather than soft-constraint methods | Constrained allocation tasks · Synthetic benchmark, Portfolio Optimization, Compute Load Distribution | constraint violations | Helped |
| Add de-biased initialization (ablation) | PASPO · Synthetic benchmark | average episode reward | Helped |
| Reverse the allocation order (ablation) | PASPO with de-biased initialization · Synthetic benchmark | average episode reward | No clear effect |
Surprising findings
- Without de-biased initialization, reversing the allocation order significantly changes performance, while with de-biasing the effect vanishes.
Limitations
- Computationally expensive for allocation tasks with many entities because each action requires a series of linear programs.
- Struggles in very high-dimensional continuous action spaces.
- Only evaluated on hard constraints; state-dependent constraints not yet handled.
Experimental setup
Datasets: NASDAQ-100 derived HMM financial environment compute load distribution environment synthetic MLP reward benchmark
Benchmarks: synthetic allocation benchmark portfolio optimization environment compute load distribution environment
Metrics: average episode reward number of constraint violations per episode
Baselines: CPO CUP IPO P3O PPO-Lagrangian OptLayer
Builds on: PPO OptLayer Dirichlet policy autoregressive factorization
Closest papers by meaning
- Augmented Lagrangian Risk-constrained Reinforcement Learning for Portfolio Optimization (Student Abstract) (AAAI 2025 · similarity 0.85)
- Online Resource Allocation With General Constraints (NeurIPS 2026 · similarity 0.85)
- ROIDICE: Offline Return on Investment Maximization for Efficient Decision Making (NeurIPS 2024 · similarity 0.84)
- Autonomous Sparse Mean-CVaR Portfolio Optimization (ICML 2024 · similarity 0.84)
- Navigating the Social Welfare Frontier: Portfolios for Multi-objective Reinforcement Learning (ICML 2025 · similarity 0.83)
- Fair and Welfare-Efficient Constrained Multi-Matchings under Uncertainty (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.