Research map / Reinforcement learning / Offline RL
NetworkGym: Reinforcement Learning Environments for Multi-Access Traffic Management in Network Simulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces NetworkGym, a ns-3-based gym-like simulator for multi-access (Wi-Fi/LTE) traffic splitting, and Pessimistic TD3 (PTD3), a TD3+BC variant with Fisher-information-based value pessimism, showing standard offline RL baselines often fail to beat heuristics.
Problem
Lack of standardized RL benchmarks in computer networking, and unclear whether offline RL methods that excel on D4RL transfer to multi-access traffic splitting between Wi-Fi and LTE.
Approach
A simulator exposing gym-like APIs over ns-3 for GMA multi-access traffic splitting, plus PTD3, which replaces TD3+BC's behavioral cloning term with a pessimistic penalty based on the Fisher-information-estimated uncertainty of Q-values.
Key results
- PTD3 averages 0.946 ± 0.043 across three datasets, the best offline average, versus 0.750 for the heuristic baseline and 0.616 for LB-SAC.
- Most off-the-shelf offline RL algorithms with default hyperparameters fail to beat the heuristic average; EDAC averages 0.120 and CQL (norm) 0.749.
- Online PPO reaches 1.214 ± 0.037, setting a soft upper bound; PTD3 on utility_logistic data reaches 1.252 with large beta, comparable to PPO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PTD3 instead of the heuristic baseline | NetworkGym multi-access traffic splitting · NetworkGym average across datasets | return per step: 0.750 ± 0.047 -> 0.946 ± 0.043 | Helped |
| Disable state normalization (ablation) | off-the-shelf offline RL algorithms · NetworkGym average across datasets | return per step: TD3+BC: 0.511 ± 0.047 -> 0.849 ± 0.045; IQL: 0.779 ± 0.047 -> 0.848 ± 0.042; CQL: 0.749 ± 0.047 -> 0.779 ± 0.056; BC: 0. | Mixed |
| Increase PTD3 β on system_default data (ablation) | PTD3 · NetworkGym system_default dataset | return per step: 0.878 ± 0.048 -> 1.017 ± 0.044 -> 0.213 ± 0.070 | Mixed |
| Increase PTD3 β on utility_logistic data (ablation) | PTD3 · NetworkGym utility_logistic dataset | return per step: 0.816 ± 0.045 -> 1.252 ± 0.063 | Helped |
| Remove the Q-value maximization term, leaving uncertainty minimization (ablation) | PTD3 · NetworkGym throughput_argmax dataset | return per step: BC (no norm) 0.751 ± 0.055; PTD3 0.770 ± 0.055 | No clear effect |
| Remove the Q-value maximization term, leaving uncertainty minimization (ablation) | PTD3 · NetworkGym system_default dataset | return per step: BC (no norm) 0.432 ± 0.063; PTD3 0.547 ± 0.059 | No clear effect |
| Remove the Q-value maximization term, leaving uncertainty minimization (ablation) | PTD3 · NetworkGym utility_logistic dataset | return per step: 0.948 ± 0.042 -> 1.252 ± 0.079 | Helped |
| Use offline RL algorithms with default state normalization instead of heuristic policies | off-the-shelf offline RL algorithms · NetworkGym average across datasets | return per step | No clear effect |
Surprising findings
- State normalization, a default feature, hurt several offline algorithms; CQL and IQL only beat heuristics when normalization was disabled.
- Purely pessimistic PTD3 (removing the Q-maximization term) outperforms behavioral cloning on the utility_logistic dataset (1.252 vs 0.948).
Limitations
- Fixed number of UEs; changing UE count requires retraining from scratch.
- Degenerate UE movement pattern and simple two-AP, one-LTE geometry are unrealistic.
- PTD3 memory scales as d x d for critic parameters, forcing small MLP critics (two layers of 64 neurons).
Experimental setup
Datasets: NetworkGym offline datasets from throughput_argmax, system_default, utility_logistic heuristics
Benchmarks: NetworkGym nqos_split multi-access traffic splitting environment
Metrics: average return per step 95% confidence interval minimum eigenvalue of feature covariance condition number
Baselines: CQL IQL TD3+BC EDAC LB-SAC SAC-N BC PPO SAC throughput_argmax system_default utility_logistic
Builds on: TD3+BC CORL Pessimistic Fitted Q-Learning ns-3 stable-baselines3
Closest papers by meaning
- URB - Urban Routing Benchmark for RL-equipped Connected Autonomous Vehicles (NeurIPS 2025 · similarity 0.83)
- Imitation Learning Backoff: Reinforcement Learning-based Channel Access for Guaranteeing Fairness (Student Abstract) (AAAI 2025 · similarity 0.83)
- Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement Learning (NeurIPS 2025 · similarity 0.83)
- QGym: Scalable Simulation and Benchmarking of Queuing Network Controllers (NeurIPS 2024 · similarity 0.83)
- Hierarchical Optimization via LLM-Guided Objective Evolution for Mobility-on-Demand Systems (NeurIPS 2025 · similarity 0.82)
- Advancing Multi-agent Traffic Simulation via R1-Style Reinforcement Fine-Tuning (ICLR 2026 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.