Research map / Multi-agent systems and games / Cooperative multi-agent RL
Distributionally Robust Multi-Agent Reinforcement Learning for Dynamic Chute Mapping
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DRMARL, combining group distributionally robust optimization with multi-agent Q-learning (VDN-style) and a contextual-bandit predictor of the worst-case induction group, to learn robust destination-to-chute mappings in Amazon sortation warehouses.
Problem
Destination-to-chute mapping in robotic sortation warehouses degrades under uncertain and shifting package induction distributions, causing package recirculation; existing MARL policies assume the deployment distribution matches training.
Approach
Group DRO over historical induction-distribution groups is embedded in MARL via a distributionally robust Bellman operator, and a contextual bandit Q-network predicts the worst-case group per state-action pair, avoiding exhaustive group evaluation.
Key results
- DRMARL reduces package recirculation rate by 79.97% on average over 21 groups versus the MARL baseline, with a 5.62% throughput increase.
- In the simplified simulator, DRMARL with the CB predictor achieves recirculation rate 0.56% versus 2.16% for MARL, and matches exhaustive worst-case search (0.55%) while training in under 300 s versus at least 2900 s.
- Exhaustive worst-case group search requires roughly 924 hours on a 64-vCPU cloud instance, while CB-based training is CPU-only.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use group-DRO training with CB-predicted worst-case induction groups instead of standard MARL | Chute mapping policy · Large-scale warehouse, 21 induction groups | Recirculation rate reduction: 79.97\% | Helped |
| use group-DRO training with CB-predicted worst-case induction groups instead of standard MARL | Chute mapping policy · Large-scale warehouse, 21 induction groups | Throughput increase: 5.62 \% | Helped |
| use group-DRO training with CB-predicted worst-case induction groups instead of standard MARL | Chute mapping policy · Large-scale warehouse, 21 induction groups | Recirculation amount reduction: 33.64 \% | Helped |
| replace CB worst-case group prediction with random group selection (ablation) | DRMARL · Simplified warehouse, 9 groups | Recirculation rate: 0.56\% \pm 0.18\% -> 1.56\% \pm 1.45\% | Hurt |
| replace CB worst-case group prediction with random group selection (ablation) | DRMARL · Simplified warehouse, 9 groups | Throughput: 11932.21 -> 11812.77 | Hurt |
| replace CB worst-case group prediction with random group selection (ablation) | DRMARL · Simplified warehouse, 9 groups | Recirculation amount: 67.79 -> 187.23 | Hurt |
| use CB prediction instead of exhaustive worst-case group search (ablation) | DRMARL · Simplified warehouse, 9 groups | Recirculation rate: 0.55\% \pm 0.13\% -> 0.56\% \pm 0.18\% | No clear effect |
| use CB prediction instead of exhaustive worst-case group search (ablation) | DRMARL · Simplified warehouse, 9 groups | Throughput: 11933.68 -> 11932.21 | No clear effect |
| use CB prediction instead of exhaustive worst-case group search (ablation) | DRMARL · Simplified warehouse, 9 groups | Recirculation amount: 66.32 -> 67.79 | No clear effect |
| use one robust policy instead of a group-specific MARL policy | Chute mapping policy · Simplified warehouse, 9 groups | Recirculation rate: 0.53\% \pm 0.14 \% -> 0.56\% \pm 0.18\% | Mixed |
| use one robust policy instead of group-specific policies | Chute mapping policy · Large-scale warehouse, 21 induction groups | Throughput increase: 9.80 \% -> 5.62 \% | Mixed |
| replace exhaustive group evaluation with CB worst-case prediction (ablation) | DRMARL training · Simplified warehouse | Training time: less than 300 seconds; at least 2900 seconds | Helped |
Surprising findings
- Group-specific MARL achieves a larger average recirculation reduction (85.42%) than DRMARL (79.97%), indicating the expected robustness trade-off.
- DRMARL shows slightly negative throughput change on two Year-4 groups (-0.99% and -5.85%) while reducing recirculation.
Limitations
- Large-scale data is proprietary (Amazon) and results are reported only as relative improvements.
- DRMARL performs slightly below group-specific MARL on in-distribution groups, a robustness trade-off.
- Bellman operator is only approximated in the large-scale setting since transitions depend on the induction variable; relative approximation error reported as under 0.57%.
Experimental setup
Datasets: Amazon historical package induction data (Years 1-4, 21 weekly groups) synthetic normal-distribution induction data (simplified simulator)
Benchmarks: Amazon robotic sortation warehouse simulator (large-scale) simplified robotic sortation simulator
Metrics: recirculation rate throughput recirculation amount training time
Baselines: MARL (regular) MARL (group-specific) DRMARL (random group selection) DRMARL (exhaustive search)
Builds on: Value Decomposition Network Deep Q-Network Group DRO Distributionally robust reinforcement learning
Closest papers by meaning
- LNS2+RL: Combining Multi-agent Reinforcement Learning with Large Neighborhood Search in Multi-agent Path Finding (AAAI 2025 · similarity 0.85)
- Single-Trajectory Distributionally Robust Reinforcement Learning (ICML 2024 · similarity 0.85)
- Robust Gymnasium: A Unified Modular Benchmark for Robust Reinforcement Learning (ICLR 2025 · similarity 0.85)
- Sample-Efficient Robust Multi-Agent Reinforcement Learning in the Face of Environmental Uncertainty (ICML 2024 · similarity 0.84)
- Time-Constrained Robust MDPs (NeurIPS 2024 · similarity 0.84)
- Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithms (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.