Research map / Reinforcement learning / Policy gradient and actor-critic

Speeding up Policy Simulation in Supply Chain RL

ICML 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Tree search and policy optimization · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Picard iteration, a parallel scheme that accelerates serial policy simulation of supply chain trajectories by batching policy evaluations and iterating over cached actions, with provable convergence in a number of iterations tied to supply-chain structure.

Problem

Simulating a single trajectory under a state-dependent neural policy is serial and slow for supply chain optimization (SCO) problems with horizons of hundreds of millions of steps, bottlenecking policy optimization.

Approach

Partition time-steps across virtual processes that each evaluate the expensive policy only on their assigned steps while using cached actions elsewhere, then iterate cache updates (Picard iteration). For fulfillment optimization, convergence occurs within a number of iterations bounded by the number of depleted nodes (at most J+1), independent of horizon T, enabling GPU batching.

Key results

• Picard achieves about 441x speedup over sequential simulation at batch size 1e4 on a 3M-order, 1M-product FO instance on a single A100.
• End-to-end policy optimization on the 3M-order instance drops from about 10 hours sequentially to under 2 minutes with Picard.
• Picard is about 88x faster than a GPU-adapted Time Warp, which achieves only about 5x over sequential simulation.

Measured effects reported in the paper

Surprising findings

• In five of eight MuJoCo environments, Picard converges to 0.1% relative RMSE in under five iterations despite T=200, suggesting convergence far below the worst-case bound.
• Uniform random partitioning of orders keeps speedup near 350x under heavy-tailed demand, whereas product-based partitioning collapses to 5x at beta=-1.0.

Limitations

• Theoretical convergence guarantee requires assumptions on the policy (inventory independence, consistency, monotonicity); violated for capacity-dependent policies, though empirically robust.
• Speedup decreases with heavy-tailed demand, and speedup depends on the ratio of transition cost to policy cost.
• Theory currently covers FO and OWMR only; MuJoCo convergence lacks theoretical basis.

Experimental setup

Datasets: synthetic fulfillment network data (J=30 nodes, up to 3M orders) MuJoCo environments via Brax

Benchmarks: synthetic fulfillment optimization instances OpenAI Gym MuJoCo environments (Brax) one warehouse multi-retailer inventory control

Metrics: speedup relative to sequential simulation relative RMSE of state trajectory runtime

Baselines: sequential policy simulation Time Warp (GPU-adapted) PPO sequential rollout

Builds on: Picard iteration Time Warp parallel discrete event simulation PPO Brax

Closest papers by meaning

• Beyond Stationarity: Convergence Analysis of Stochastic Softmax Policy Gradient Methods (ICLR 2024 · similarity 0.84)
• Adaptive-Gradient Policy Optimization: Enhancing Policy Learning in Non-Smooth Differentiable Simulations (ICML 2024 · similarity 0.82)
• Convergence Analysis of Policy Gradient Methods with Dynamic Stochasticity (ICML 2025 · similarity 0.82)
• Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modeling (NeurIPS 2025 · similarity 0.82)
• Improving Value Estimation Critically Enhances Vanilla Policy Gradient (ICML 2025 · similarity 0.81)
• Relative Policy-Transition Optimization for Fast Policy Transfer (AAAI 2024 · similarity 0.81)