Research map / Reinforcement learning / Offline RL
NondBREM: Nondeterministic Offline Reinforcement Learning for Large-Scale Order Dispatching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NondBREM, an offline deep RL framework for ride-hailing order dispatching that combines a similarity-constrained nondeterministic BCQ action set with random ensemble mixture multi-head Q-networks, trained on logged data.
Problem
Order dispatching in ride-hailing requires online RL that is costly and unsafe; offline RL must handle extrapolation error and a dynamic, nondeterministic action space of order-vehicle matches.
Approach
Extends BCQ to a nondeterministic action space by selecting logged orders whose cosine similarity to VAE-generated candidate actions is high, adds a KL term for vehicle-order distribution balance, and uses REM multi-head Q-networks for robustness.
Key results
- NondBREM achieves a dispatch score of 2043643.57 and +3.76% ORR improvement over the IL baseline, the best among compared methods.
- NondBREM's offline-trained policy outperforms online-trained PolarB (+2.52%) and KL-Based (+3.12%).
- Using only 25% of data, NondBREM still reaches 2012126.03 dispatch score, above NondBCQ and KL-Based offline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use NondBREM instead of IL | Offline order dispatching · Ride-hailing simulator, two-week evaluation | ORRIR: 0 -> +3.76 | Helped |
| Use NondBREM instead of PolarB | Order dispatching · Ride-hailing simulator, two-week evaluation | dispatch score: 2024634.79 -> 2043643.57 | Helped |
| Use NondBREM instead of KL-Based offline | Offline order dispatching · Ride-hailing simulator, two-week evaluation | dispatch score: 1864928.20 -> 2043643.57 | Helped |
| Add the REM multi-head value network (ablation) | NondBCQ · Ride-hailing simulator, two-week evaluation | dispatch score: 2005392.15 -> 2043643.57 | Helped |
| Add the REM multi-head value network (ablation) | NondBCQ · Ride-hailing simulator, two-week evaluation | ORRIR: +2.61 -> +3.76 | Helped |
| Use β = 1, removing action-set restriction (ablation) | NondBREM · Ride-hailing simulator, online decision process | normalized cumulative reward | Hurt |
| Set β to β3 = min(max(n′/n, 0.9), 1) (ablation) | NondBREM · Ride-hailing simulator, online decision process | normalized cumulative reward | Helped |
| Train with 50% rather than 25% of the data (ablation) | NondBREM · Ride-hailing simulator | dispatch score: 2012126.03 -> 2024210.18 | Helped |
| Train with 50% rather than 25% of the data (ablation) | NondBCQ · Ride-hailing simulator | dispatch score: 1981308.81 -> 1989275.19 | Helped |
| Train with 50% rather than 25% of the data (ablation) | KL-Based offline · Ride-hailing simulator | dispatch score: 1831410.23 -> 1834626.25 | Helped |
Surprising findings
- KL-Based trained offline performs worse than the online IL baseline (-2.35% ORR).
- Setting the action set to the full environment order count (beta=1) degrades performance, as the method reduces to standard RL with higher extrapolation error.
Limitations
- Evaluation relies on a sampling-based simulator with proprietary ride-hailing data from one city, limiting reproducibility.
- Inference is about 20x slower than the IL baseline due to the VAE component.
Experimental setup
Datasets: real-world ride-hailing order data vehicle GPS data
Benchmarks: real-world ride-hailing dispatch simulator (2 weeks held-out)
Metrics: dispatch score (total driver income) ORR improvement rate MSE inference time
Baselines: IL TVal KL-Based PolarB NondBCQ
Builds on: Batch-Constrained Q-learning Random Ensemble Mixture Double DQN KL-based order dispatching
Closest papers by meaning
- One Step is Enough: Multi-Agent Reinforcement Learning Based on One-Step Policy Optimization for Order Dispatch on Ride-Sharing Platforms (NeurIPS 2026 · similarity 0.88)
- GARLIC: GPT-Augmented Reinforcement Learning with Intelligent Control for Vehicle Dispatching (AAAI 2025 · similarity 0.87)
- Hierarchical Optimization via LLM-Guided Objective Evolution for Mobility-on-Demand Systems (NeurIPS 2025 · similarity 0.86)
- i-Rebalance: Personalized Vehicle Repositioning for Supply Demand Balance (AAAI 2024 · similarity 0.84)
- In-Dataset Trajectory Return Regularization for Offline Preference-based Reinforcement Learning (AAAI 2025 · similarity 0.82)
- Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning (ICLR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.