Research map / Multi-agent systems and games / Cooperative multi-agent RL
Learning to Steer Markovian Agents under Model Uncertainty
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies steering Markovian agents (whose policy updates depend only on current policy and reward) toward desired policies via steering rewards in a finite-horizon non-episodic setting under model uncertainty, proposing a history-dependent steering objective with belief-state and FETE (explore-then-exploit) algorithms.
Problem
Steering multi-agent learning dynamics toward desired policies via additional rewards when the agents' learning dynamics are unknown, with only one finite-horizon episode of interaction with real agents.
Approach
Model Markovian agents (recency-biased, policy updates depend only on current policy and modified reward) and formulate steering as a non-episodic RL problem over a finite model class, learning history-dependent steering strategies via a belief-state lifted MDP for small model classes and a First-Explore-Then-Exploit framework with MLE identification for large ones.
Key results
- In the Stag Hunt with model uncertainty (mu in {0.7,1.0}), the belief-state strategy achieves epsilon-steering gap with probability 0.87 in f_0.7 and 0.99 in f_1.0, while the non-belief baseline psi*_1.0 achieves only 0.13 in f_0.7.
- In the 10-player avaricious-agent setting with |F|=3^10, FETE's steering gap and cost are described as competitive with the Oracle that knows f*, and show significant advantage over FETE-RE (SIAR-MPC adaptation) in steering gap.
- Theoretically, steering strategies with epsilon-steering gap exist for NPG/PMD agents under sufficient conditions (given enough steps T and steering budget U_max).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a belief-state steering strategy instead of the μ=1.0 strategy designed as if the model were known | NPG agents in the Stag Hunt · Stag Hunt, f_{0.7} | p(Δψ,T ≤ ε): 0.13 ± 0.02 -> 0.87 ± 0.05 | Helped |
| Use a belief-state steering strategy instead of the μ=1.0 strategy designed as if the model were known | NPG agents in the Stag Hunt · Stag Hunt, f_{0.7} | Cψ,T: 7.6 ± 0.2 -> 10.5 ± 0.4 | Hurt |
| Use a belief-state steering strategy instead of the μ=0.7 strategy designed as if the model were known | NPG agents in the Stag Hunt · Stag Hunt, f_{1.0} | Cψ,T: 8.2 ± 0.2 -> 6.1 ± 0.3 | Helped |
| Use strategic exploration that learns from identification success instead of noise-based random exploration | FETE in the avaricious-agent setting · Cooperative game, |F| = 3^10 | exploration efficiency | Helped |
| Use FETE instead of FETE-RE with random exploration | Steering 10 avaricious agents · Cooperative game, f1, f2, and f3 | steering gap | Helped |
Surprising findings
- A strategy trained for the wrong model (psi*_1.0 deployed on f_0.7) fails to reach the desired accuracy, showing that ignoring model uncertainty is harmful.
Limitations
- Only agents with Markovian (non-regret, memoryless) updates are covered; non-Markovian agents left to future work.
- Steering reward budget constraints are not considered.
- Experiments use small or simulated games; FETE's gap is only described as competitive with the Oracle.
Experimental setup
Benchmarks: Normal-form Stag Hunt Grid-world Stag Hunt Matching Pennies 10-player cooperative avaricious-agent game
Metrics: steering gap steering cost probability of epsilon-steering gap identification success rate
Baselines: SIAR-MPC FETE-RE (random exploration variant) Oracle (known f*) psi*_0.7 psi*_1.0
Builds on: Natural policy gradient Policy mirror descent PPO SIAR-MPC
Closest papers by meaning
- Learning in Causal Markov Games (NeurIPS 2026 · similarity 0.87)
- Multi-Agent Meta-Reinforcement Learning: Sharper Convergence Rates with Task Similarity (NeurIPS 2023 · similarity 0.87)
- Reinforcement Learning in Presence of Discrete Markovian Context Evolution (ICLR 2022 · similarity 0.87)
- Reinforcement Learning in Reward-Mixing MDPs (NeurIPS 2021 · similarity 0.87)
- Improving Sample Efficiency of Model-Free Algorithms for Zero-Sum Markov Games (ICML 2024 · similarity 0.87)
- Incentivize without Bonus: Provably Efficient Model-based Online Multi-agent RL for Markov Games (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.