Research map / Large language models / Agents, code and math
Intelligence at the Edge of Chaos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Pretrains GPT-2 models on next-state prediction of individual elementary cellular automata (ECA) rules, then freezes them and tests downstream ARC-style reasoning, chess move prediction, and Nim, finding that rules with more complex (Wolfram Class IV/III) dynamics yield better downstream performance.
Problem
Whether the complexity of the rule-based system used for pretraining affects the general capabilities (intelligence) that transformers acquire, and whether intelligence can emerge from exposure to simple but complex-behaving synthetic systems without intelligent data.
Approach
Pretrain small GPT-2 variants from scratch on binary spatiotemporal data from single ECA rules spanning Wolfram's four classes, freeze the transformer body, and measure how quickly/accurately it adapts to downstream tasks; correlate performance with multiple complexity measures (Lempel-Ziv, compression, Lyapunov, Krylov).
Key results
- Models pretrained on more complex ECA data (higher Lempel-Ziv complexity) show significant positive correlation with downstream efficiency on ARC-Easy (Pearson 0.7322, p=4.93e-18) and NIM (Pearson 0.6314).
- Wolfram Class III and IV rules outperform Class I and II rules on reasoning and chess tasks, with Class IV especially strong on chess move prediction.
- Models trained on complex rules attend more to historical states (r=0.66 between data complexity and average attention), while uniform/periodic rule models attend little to history.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pretrain on higher Lempel-Ziv-complexity ECA data | GPT-2 with frozen pretrained layers and trained downstream projections · ARC-Easy | Efficiency (inverse epochs to 80% accuracy): Pearson correlation 0.7322; distance correlation 0.7163 | Helped |
| Pretrain on higher Lempel-Ziv-complexity ECA data | GPT-2 with frozen pretrained layers and trained downstream projections · ARC-Hard | Efficiency (inverse epochs to 80% accuracy): Pearson correlation 0.3896; distance correlation 0.4444 | Helped |
| Pretrain on higher Lempel-Ziv-complexity ECA data | GPT-2 with frozen pretrained layers and trained downstream projections · Chess | Move prediction accuracy: Pearson correlation 0.4371; distance correlation 0.4329 | Helped |
| Pretrain on higher Lempel-Ziv-complexity ECA data | GPT-2 with frozen pretrained layers and trained downstream projections · NIM | Efficiency (inverse epochs to 20% accuracy): Pearson correlation 0.6314; distance correlation 0.7108 | Helped |
| Pretrain on temporally ordered ECA data rather than randomly shuffled states (ablation) | GPT-2 · ARC-Easy | Efficiency (inverse epochs to 80% accuracy) | Helped |
| Predict 1 future step instead of 5 (ablation) | GPT-2 pretrained on ECA rules · ECA prediction | Prediction performance | Helped |
| Pretrain on Class IV ECA rules rather than Class I or II rules | GPT-2 with frozen pretrained layers and trained downstream projections · Chess | Move prediction accuracy | Helped |
| Pretrain on highly chaotic Class III rules rather than rules with structured complexity | GPT-2 with frozen pretrained layers and trained downstream projections · ARC-Hard and Chess | Downstream performance | Hurt |
Surprising findings
- Models predicting only the next step learned non-trivial solutions and outperformed models predicting five steps ahead.
- Some Class III chaotic rules (e.g., 105, 146, 150) performed poorly, indicating an inverted-U relationship with complexity.
Limitations
- Downstream evaluation freezes the transformer body and trains only input/output projections, so results may not reflect full fine-tuning.
- Correlation analyses are over a limited set of ECA rules and a few downstream tasks; the link to general intelligence is conjectural.
- Performance on the chess and Nim tasks is compared with a randomly initialized baseline only, with no comparison to models pretrained on natural or human-generated data.
Experimental setup
Datasets: elementary cellular automata simulations (ECA rules) ARC-inspired synthetic reasoning sequences Lichess Elite database (chess games) synthetic Nim games
Benchmarks: ARC-inspired easy reasoning task ARC-inspired hard reasoning task chess move prediction (Lichess Elite) Nim optimal move prediction
Metrics: efficiency (1/epochs to 80% accuracy) move prediction accuracy Pearson correlation distance correlation attention scores
Baselines: randomly initialized transformer
Builds on: GPT-2 ARC Wolfram elementary cellular automata classification
Closest papers by meaning
- Emergence of Physical Intelligence via Controllable Information Production (NeurIPS 2026 · similarity 0.86)
- Creativity or Brute Force? Using Brainteasers as a Window into the Problem-Solving Abilities of Large Language Models (NeurIPS 2025 · similarity 0.84)
- A Percolation Model of Emergence: Analyzing Transformers Trained on a Formal Language (ICLR 2025 · similarity 0.84)
- AlphaZero Neural Scaling and Zipf's Law: a Tale of Board Games and Power Laws (NeurIPS 2025 · similarity 0.84)
- Neuron-based Multifractal Analysis of Neuron Interaction Dynamics in Large Models (ICLR 2025 · similarity 0.84)
- Emergent Communication for Numerical Concepts Generalization (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.