Research map / Multi-agent systems and games / Cooperative multi-agent RL
Multi-Agent Domain Calibration with a Handful of Offline Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Madoc, which calibrates simulator physics parameters to match target-domain offline trajectories by casting it as a bandit RL objective with discriminator rewards, solved via VAE-based grouping and cooperative multi-agent RL (DOP + SAC).
Problem
Policies trained in a source simulator degrade when deployed in a target domain with different dynamics; existing offline domain calibration relies on evolutionary/sampling black-box search that is sample-inefficient when many physics parameters must be calibrated.
Approach
Derive a bandit RL reward for calibration from a KL trajectory-matching objective using a pair of binary discriminators (reward model); model each physics parameter as a calibration agent, cluster parameters into groups via a VAE embedding and k-means, and train the group agents cooperatively with value decomposition (DOP) and SAC-style actors.
Key results
- On D4RL, Madoc averages 88.3 normalized return versus 74.6 for H2O and 71.2 for MOREC in the target domain.
- On NeoRL, Madoc averages 70.8 normalized return versus 57.2 for the ablation Madoc-S and 48.4 for OTED.
- Without automatic grouping, Madoc's HalfCheetah NeoRL performance drops significantly; Madoc-VDN and Madoc-ISAC underperform Madoc on HalfCheetah D4RL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Madoc instead of H2O | offline domain calibration and transfer on D4RL · D4RL average | normalized average return: 74.6 -> 88.3 | Helped |
| use Madoc instead of OTED | offline domain calibration and transfer on NeoRL · NeoRL average | normalized average return: 48.4 -> 70.8 | Helped |
| add multi-agent decomposition and automatic grouping instead of using a single calibration agent (ablation) | Madoc · D4RL average | normalized average return: 68.9 -> 88.3 | Helped |
| add multi-agent decomposition and automatic grouping instead of using a single calibration agent (ablation) | Madoc · NeoRL average | normalized average return: 57.2 -> 70.8 | Helped |
| remove automatic grouping, assigning one agent to each parameter (ablation) | Madoc · HalfCheetah NeoRL | average return | Hurt |
| replace DOP with VDN for value decomposition (ablation) | Madoc · HalfCheetah D4RL medium, medium-replay, and medium-expert | normalized average return: 91.9± 7.7 -> 89.9± 6.9; 95.7± 9.9 -> 92.7± 7.1; 96.9± 5.3 -> 88.2± 5.7 | Hurt |
| replace value decomposition with independent learning (ablation) | Madoc · HalfCheetah D4RL medium, medium-replay, and medium-expert | normalized average return: 91.9± 7.7 -> 81.1±14.7; 95.7± 9.9 -> 75.0±13.1; 96.9± 5.3 -> 75.2±16.1 | Hurt |
| add behavior-cloning constraints during final policy training (ablation) | Madoc · HalfCheetah D4RL medium, medium-replay, and medium-expert | normalized average return: 91.9±7.7 -> 88.9±4.6; 95.7±9.9 -> 90.1±4.8; 96.9±5.3 -> 97.2±3.3 | Mixed |
| vary the number of parameter groups (ablation) | Madoc · HalfCheetah D4RL medium-replay | normalized average return | No clear effect |
| vary the calibration-policy temperature coefficient (ablation) | Madoc · HalfCheetah D4RL medium-replay | normalized average return | No clear effect |
Surprising findings
- Independent learning (Madoc-ISAC) and VDN variants are clearly worse than DOP-based Madoc, suggesting non-stationarity matters even in bandit-style calibration.
- Madoc has higher variance across seeds than offline baselines, attributed to exploration in the source domain; adding BC after calibration reduces variance at a slight mean cost.
Limitations
- Reward model from a handful of offline data may not generalize well to high-dimensional vision tasks, as acknowledged by the authors.
- Higher variance across random seeds than pure offline baselines.
- Requires online interaction with the simulator (source domain) during calibration, with extra compute and memory compared to CQL.
Experimental setup
Datasets: D4RL HalfCheetah, Hopper, Walker2d, Ant (medium, medium-replay, medium-expert) NeoRL HalfCheetah, Hopper, Walker2d (low, medium, high)
Benchmarks: D4RL NeoRL
Metrics: normalized average return mean absolute calibration error 95% confidence interval over 6 seeds
Baselines: DROPO DROID OTED H2O DR+BC CQL MOREC Madoc-S
Builds on: DARC OTED DROPO DROID DOP SAC
Closest papers by meaning
- Cross-Domain Policy Adaptation by Capturing Representation Mismatch (ICML 2024 · similarity 0.88)
- Localized Dynamics-Aware Domain Adaption for Off-Dynamics Offline Reinforcement Learning (NeurIPS 2026 · similarity 0.88)
- Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning (AAAI 2024 · similarity 0.86)
- Off-Dynamics Reinforcement Learning via Domain Adaptation and Reward Augmented Imitation (NeurIPS 2024 · similarity 0.86)
- Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learning (ICML 2024 · similarity 0.86)
- The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.