Research map / Architectures and efficiency / Attention and transformers

Chess-World-Model: A 10M-Game Benchmark for Exact State Tracking from Chess Move Sequences

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: State space and recurrent sequence models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Chess-World-Model, a 10M-game benchmark where sequence models predict the exact FEN-style chess state after each move prefix, evaluated on held-out real games and uniformly random legal play, comparing a Transformer with expressive recurrent models (SLiCE, Mamba-3, Gated DeltaNet).

Problem

Exact state tracking in sequence models is hard to evaluate in realistic domains: synthetic state-tracking benchmarks are small and regular, and in-distribution evaluation can hide failures that scale conceals.

Approach

Treat chess as an exactly checkable learned game engine: from a UCI move sequence, predict the full 75-label chess state at every prefix, and add a random-uniform legal-play split as an out-of-distribution test of rule-consistent state updates.

Key results

• Recurrent models strongly outperform the Transformer at d128/l1 and d256/l2; e.g., at d256/l2 real-game ExactState is 74.44% for Transformer vs 96.57% SLiCE, 98.13% Mamba-3, 99.12% Gated DeltaNet.
• Held-out real-game ExactState saturates near 99%+ by d384/l4, while random-uniform ExactState remains discriminative; the best random-uniform ExactState is 81.13% (Mamba-3 at d512/l6).
• Less expressive transitions reduce random-uniform performance for SLiCE, Mamba (Mamba-2), and Gated DeltaNet; diagonal SLiCE has very large OOD cross-entropy (e.g., 4.43e17 at d256/l2).

Surprising findings

• Diagonal SLiCE shows numerical instability on random-uniform data, with cross-entropy reaching extreme values despite strong held-out real-game performance.
• Models near-perfect on held-out real games (~99%+ ExactState) can still reach only ~33-81% ExactState on random-uniform games.

Limitations

• Chess removes perceptual uncertainty, stochasticity, partial observability and continuous control, so it is a narrow test of state updates.
• Random-uniform split is only one form of distribution shift; other agents, adversarial sequences, longer games, or rare rules are not tested.
• Comparison limited to a shared supervised interface and a single training protocol with one seed in the sweep.

Experimental setup

Datasets: Lichess Open Database (March 2025 training, April 2025 held-out test) random-uniform legal self-play games

Benchmarks: Chess-World-Model held-out real-game test Chess-World-Model random-uniform test

Metrics: ExactState accuracy labelwise state accuracy trajectory exactness cross-entropy loss

Baselines: causal Transformer (LLaMA-like) diagonal SLiCE Mamba-2 Gated DeltaNet with positive eigenvalues

Builds on: Mamba-3 Gated DeltaNet SLiCE Mamba-2

Closest papers by meaning

• Amortized Planning with Large-Scale Transformers: A Case Study on Chess (NeurIPS 2024 · similarity 0.89)
• The Illusion of State in State-Space Models (ICML 2024 · similarity 0.87)
• Implicit Search via Discrete Diffusion: A Study on Chess (ICLR 2025 · similarity 0.86)
• Evidence of Learned Look-Ahead in a Chess-Playing Neural Network (NeurIPS 2024 · similarity 0.86)
• A Causal World Model Underlying Next Token Prediction: Exploring GPT in a Controlled Environment (ICML 2025 · similarity 0.86)
• Rethinking State Tracking in Recurrent Models Through Error Control Dynamics (NeurIPS 2026 · similarity 0.85)