Research map / Image generation / Autoregressive and latent generation
Transcendence: Generative Models Can Outperform The Experts That Train Them
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that a chess transformer trained on human Lichess games can outperform the best human players in its data when sampled at low temperature, with theory linking low-temperature sampling to majority voting over experts.
Problem
Whether generative models trained by imitation of human experts can surpass the best expert in their training data (transcendence), and under what conditions.
Approach
Formalizes transcendence and proves that pure imitation (cross-entropy mixture of experts) cannot exceed the best expert, but low-temperature sampling approximates an argmax majority vote over experts that can exceed every individual expert, denoising idiosyncratic errors.
Key results
- ChessFormer 1000 and ChessFormer 1300 reach about 1500 Glicko-2 at temperature 0.001, surpassing the maximum rating in their training data, while ChessFormer 1500 does not transcend.
- Low temperature (0.001) raises expected win probability over temperature 1.0 by 2.15 ± 0.17 percentage points and top-1 accuracy from 22.61% to 29.61%.
- Gains are concentrated in a small set of states, with some states improving by over 5% in win probability.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Lower sampling temperature from 1.0 to 0.001 (ablation) | ChessFormer 1000 · ChessFormer 1000 games | expected win probability (%): 37.80 ± 0.87 -> 39.95 ± 0.92 | Helped |
| Lower sampling temperature from 1.0 to 0.75 (ablation) | ChessFormer 1000 · ChessFormer 1000 games | expected win probability (%): 37.80 ± 0.87 -> 38.79 ± 0.90 | Helped |
| Lower sampling temperature from 1.0 to 0.001 (ablation) | ChessFormer 1000 · ChessFormer 1000 games | Top 1 accuracy (%): 22.61 ± 0.86 -> 29.61 ± 1.43 | Helped |
| Lower sampling temperature from 1.0 to 0.001 (ablation) | ChessFormer 1000 · ChessFormer 1000 games | Top 3 accuracy (%): 44.00 ± 9.96 -> 54.26 ± 1.57 | Helped |
| Lower sampling temperature from 1.0 to 0.001 (ablation) | ChessFormer 1000 · ChessFormer 1000 games | Top 5 accuracy (%): 56.27 ± 0.93 -> 66.86 ± 1.47 | Helped |
Surprising findings
- ChessFormer 1500 fails to transcend, which the authors attribute to limited diversity in the 1000-1500 rating band.
- Imitation learning alone provably cannot exceed the best expert; transcendence requires low-temperature sampling.
Limitations
- Theory assumes idealized infinite data, uniform expert sampling, and shared input distribution across experts.
- Chess results are against Stockfish levels on Lichess; the paper does not establish transcendence in natural language or other broad settings beyond limited SQuAD experiments.
- Transcendence relies on test states matching training distribution; the authors note extension to composition or reasoning needs other assumptions.
Experimental setup
Datasets: Lichess open database SQuAD 2.0
Benchmarks: Stockfish-based Glicko-2 chess rating SQuAD 2.0
Metrics: Glicko-2 rating expected win probability (Stockfish eval) top-k move accuracy exact match F1 semantic match
Baselines: Stockfish 1 Stockfish 3 Stockfish 5 Maia
Builds on: Glicko-2 Decision Transformer Performance Difference Lemma Maia
Closest papers by meaning
- The Generative AI Paradox: “What It Can Create, It May Not Understand” (ICLR 2024 · similarity 0.85)
- The two clocks and the innovation window: When and how generative models learn rules (NeurIPS 2026 · similarity 0.85)
- Evaluating the World Model Implicit in a Generative Model (NeurIPS 2024 · similarity 0.85)
- No Free Lunch: Fundamental Limits of Learning Non-Hallucinating Generative Models (ICLR 2025 · similarity 0.84)
- A Causal World Model Underlying Next Token Prediction: Exploring GPT in a Controlled Environment (ICML 2025 · similarity 0.84)
- Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Tasks (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.