Research map / Large language models / Agents, code and math
Out-of-Distribution Evaluation of Rule-Based and Strategic Reasoning in Chess Transformers
Anna Mészáros, Patrik Reizinger, Ferenc Huszár
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| move from ID puzzles to OOD puzzles | Transformer · Puzzles | Stockfish top1 accuracy (%): 70.50 -> 67.70 | No clear effect |
| move from ID puzzles to OOD puzzles | Transformer · Puzzles | puzzle sequence accuracy (%): 58.80 -> 54.70 | No clear effect |
| test on boards with more pieces than allowed | Transformer · More pieces | Stockfish top1 accuracy (%): 56.30 -> 30.49 | Hurt |
| test on boards with same-colored bishops | Transformer · Same color | Stockfish top1 accuracy (%): 56.30 -> 30.60 | Hurt |
| test on Chess960 starting positions | Transformer · Chess960 starting pos. | Stockfish top1 accuracy (%): 56.30 -> 22.73 | Hurt |
| test on Knights&Rooks extreme boards | Transformer · Knights&Rooks | legal move accuracy (%): 100 -> 90.20 | Hurt |
| test on Knights&Rooks extreme boards | Transformer · Knights&Rooks | Stockfish top1 accuracy (%): 56.30 -> 02.60 | Hurt |
| play Horde instead of Standard chess (Lichess) | Transformer · Lichess Elo | Lichess Elo: 1550 -> 1178 | Hurt |
| play Chess960 instead of Standard chess (Lichess) | Transformer · Lichess Elo | Lichess Elo: 1550 -> 1571 | No clear effect |
| play Chess960 instead of Standard chess against Stockfish | Transformer · Tournament | relative Elo: 88 -> 110 | Hurt |
| play Horde as Black instead of White | Transformer · Horde legal move accuracy | legal move accuracy (%): 94.74 -> 97.18 | Improved |
| play Horde as Black instead of White | Transformer · Horde games vs Fairy-Stockfish | games won: 31 -> 76 | Improved |
Closest papers by meaning
- Amortized Planning with Large-Scale Transformers: A Case Study on Chess (NeurIPS 2024 · similarity 0.90)
- Mastering Board Games by External and Internal Planning with Language Models (ICML 2025 · similarity 0.87)
- Human-Aligned Chess With a Bit of Search (ICLR 2025 · similarity 0.87)
- Chess-World-Model: A 10M-Game Benchmark for Exact State Tracking from Chess Move Sequences (NeurIPS 2026 · similarity 0.87)
- Designing Skill-Compatible AI: Methodologies and Frameworks in Chess (ICLR 2024 · similarity 0.86)
- The Algorithm Is Not the Behavior: Learned Priors Override Look-Ahead in a Chess-Playing Neural Network (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
