Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Designing Skill-Compatible AI: Methodologies and Frameworks in Chess
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a framework and three methods (Tree, Expector, Attuned) for chess AI that stays strong when paired with much weaker partners (maia 1100), evaluated in two collaborative chess variants, Stochastic Tag Team and Hand and Brain.
Problem
Strong AI agents often must interact with much weaker agents (humans or heuristics), but raw superhuman strength does not guarantee the AI's actions are usable by weaker partners; the paper asks how to build and measure skill-compatibility.
Approach
Define skill-compatibility as the ability of a weaker agent to take over from any point in a game. Two chess frameworks (random stochastic hand-off and piece-then-move hand and brain) are used to evaluate three ways of building senior agents: MCTS over a human-like model (Tree), short-horizon expectation maximization using junior models (Expector), and fine-tuning a strong engine on junior-involve
Key results
- In the STT framework, Expector (exp_t) achieves a 66.5% win-share against Leela-led teams with Maia 1100 juniors, while in HB the Tree agent scores 60.0%.
- Focal agents lose heavily to Leela in regular chess (Tree <1% win-share, Attuned 12.5%, Expector 14.0% in STT), showing they are weaker in raw chess strength.
- Playing Maia alone as senior/brain scores only 7.5% (STT) and 27.5% (HB) against the Leela team.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use skill-compatible Tree, Expector, or Attuned seniors with Maia juniors instead of Maia alone as the focal senior | STT · STT vs Leela + Maia | win-share (%): 56.5 vs 7.5; 66.5 vs 7.5; 55.0 vs 7.5 | Helped |
| Use skill-compatible Tree, Expector, or Attuned brains with Maia hands instead of Maia alone as focal | HB · HB vs Leela + Maia | win-share (%): 60.0 vs 27.5; 56.5 vs 27.5; 55.0 vs 27.5 | Helped |
| Use Expector rather than Tree or Attuned for the STT focal senior | STT vs Leela + Maia · STT | win-share (%): 56.5 -> 66.5; 55.0 -> 66.5 | Helped |
| Use Tree rather than Expector or Attuned for the HB focal brain | HB vs Leela + Maia · HB | win-share (%): 56.5 -> 60.0; 55.0 -> 60.0 | Helped |
| Fine-tune Attuned on Leela moves only, excluding Maia policy moves (ablation) | Attuned (STT) · STT with Maia1100 junior | win-share (%): 55.0 % ±0.5 -> 52% ± 0.5 | Hurt |
| Reduce Expector's look-ahead to one ply, removing the helping objective (ablation) | Expector (STT) · STT with Maia1100 junior | win-share (%): 66.5% -> 62.5% ± 1 | Hurt |
| Optimize Expector only to help its own Maia junior (ablation) | Expector (STT) · STT with Maia1100 junior | win-share (%): 66.5% -> 57.0% ± 1 | Hurt |
| Use a Maia1900-designed Tree with a Maia1100 junior instead of a Maia1100-designed Tree | Tree · STT with Maia1100 junior | win-share (%): 56.5±0.5 -> 51.0±0.5 | Hurt |
| Use a Maia1100-designed Expector with a Maia1900 junior instead of its intended Maia1100 junior | Expector (STT) · STT with Maia1900 junior | win-share (%): 66.5±0.5 -> 53.0 ± 0.5 | Hurt |
| Use a Maia1100-designed Attuned with an sfw junior instead of its intended Maia1100 junior | Attuned (STT) · STT with sfw junior | win-share (%): 55.0±0.5 -> 44.0±1.0 | Hurt |
Surprising findings
- Tree agent uses MCTS over a Maia model unrelated to Leela's weights yet outperforms the fine-tuned Attuned agent in both frameworks despite being much weaker in regular chess.
- Attuned agents trained on Maia-free Leela moves perform worse in STT than the default version, suggesting training the policy on Maia moves helps.
Limitations
- Evaluations use only Maia engines as human proxies; no experiments with real human chess players.
- Expector is computationally expensive and hard to extend to continuous environments; methods are chess-specific in part.
- Cross-style generalization fails: agents trained with Maia 1100 do not generalize to a weakened Stockfish junior, and cross-skill compatibility is asymmetric.
Experimental setup
Datasets: self-play games of Leela and Maia teams in STT and HB frameworks Maia human game data (Lichess 1100-rated and 1900-rated players) Lichess player-specific Maia models (23 players)
Benchmarks: STT framework games vs Leela with Maia 1100 juniors HB framework games vs Leela with Maia 1100 hands
Metrics: win-share (W + D/2)/n average win probability loss per move standard error
Baselines: Leela Chess Zero (128x10 net, 1500 nodes) Maia 1100 playing as senior Maia 1900 (cross-skill) Stockfish-based weakened junior (sfw)
Builds on: AlphaZero Leela Chess Zero Maia
Closest papers by meaning
- Human-Aligned Chess With a Bit of Search (ICLR 2025 · similarity 0.91)
- Maia-2: A Unified Model for Human-AI Alignment in Chess (NeurIPS 2024 · similarity 0.91)
- Amortized Planning with Large-Scale Transformers: A Case Study on Chess (NeurIPS 2024 · similarity 0.86)
- Melting Pot Contest: Charting the Future of Generalized Cooperative Intelligence (NeurIPS 2024 · similarity 0.86)
- An Evolutionary Perspective on AI Alignment (Student Abstract) (AAAI 2025 · similarity 0.85)
- Mastering Board Games by External and Internal Planning with Language Models (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.