Research map / Reinforcement learning / Visual and representation RL
QGym: Scalable Simulation and Benchmarking of Queuing Network Controllers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces QGym, an open-sourced event-driven queuing network simulator and benchmark suite comparing classical queuing policies with PPO variants across 20 instances.
Problem
Queuing network control is hard for RL due to continuous-time dynamics, high stochasticity, and long horizons with exploding delays; there is no common benchmark comparing RL and classical queuing policies on realistic instances.
Approach
An event-driven, OpenAI Gym-compatible simulator supporting general non-exponential arrivals/services and job-level tracking, plus PPO variants with work-conserving action parameterization (masking empty queues) that stabilizes RL on queuing control.
Key results
- PPO-WC achieves a 77% pairwise win rate against traditional queuing policies across instances, while vanilla PPO fails to stabilize in every setting.
- Under hyperexponential service times on reentrant L=2, PPO-WC reduces holding cost by about 21% relative to baselines.
- On the large hospital network (N=497 servers), PPO-WC (about 2.3E+4) is far worse than c-mu rule (4.4E+2), indicating RL sample inefficiency at scale.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add work-conserving probability masking to PPO (ablation) | PPO · Queuing benchmark instances | pair-wise win rate against traditional policies: 77% | Helped |
| Add behavior-cloning warm-up from a MaxWeight-style policy (ablation) | PPO · Queuing benchmark instances | training stability and time-average total queue length | Helped |
| Use vanilla PPO without work-conserving masking or behavior-cloning warm-up (ablation) | PPO · Queuing benchmark instances | network stability | Hurt |
| Use work-conserving A2C instead of work-conserving PPO (ablation) | PPO-WC · Queuing benchmark instances | performance | No clear effect |
| Apply PPO-WC to hyperexponential rather than exponential reentrant event times | PPO-WC · Reentrant networks | relative reduction in holding costs: around 21% with L = 2 | Helped |
| Scale the network to larger, more realistic instances | PPO-WC · Hospital network | time-average total queue length: 2.3\text{E+}4 \pm 7.4; 4.4\text{E+}2 \pm 0.0 | Hurt |
Surprising findings
- Vanilla PPO fails to stabilize the queuing network in every tested setting, with queue lengths exploding to 10^3-10^5.
- Behavior cloning of a MaxWeight-style policy improves PPO training but still does not match classical queuing policies.
Limitations
- RL policies perform poorly on large realistic networks such as the hospital instance.
- Time-varying service rates are not yet supported in the simulator.
- Only holding-cost objective is used as a representative reward.
Experimental setup
Datasets: simulated queuing instances (hospital, input-queue switch, reentrant, five-by-five, criss-cross, N-model)
Benchmarks: QGym criss-cross QGym reentrant (exponential and hyperexponential) QGym parallel server (N-model, five-by-five) QGym real-world (input switch, hospital)
Metrics: time-averaged total queue length standard error pairwise win rate
Baselines: c-mu rule MaxWeight maximum pressure fluid policy PPO PPO-BC A2C
Builds on: PPO OpenAI Gym MaxWeight maximum pressure policy
Closest papers by meaning
- NetworkGym: Reinforcement Learning Environments for Multi-Access Traffic Management in Network Simulation (NeurIPS 2024 · similarity 0.83)
- LineFlow: A Framework to Learn Active Control of Production Lines (ICML 2025 · similarity 0.82)
- Queueing Matching Bandits with Preference Feedback (NeurIPS 2024 · similarity 0.81)
- Decentralized Scheduling with QoS Constraints: Achieving O(1) QoS Regret of Multi-Player Bandits (AAAI 2024 · similarity 0.81)
- DynaSchedBench: Calibrated Dynamic Scheduling Benchmarks and Observability Paradox in LLM-based Scheduling Agents (ICML 2026 · similarity 0.81)
- The Complexity of Optimizing Atomic Congestion (AAAI 2024 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.