Research map / Large language models / Language, safety and interpretability
Fractal Patterns May Illuminate the Success of Next-Token Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
The paper argues language is self-similar and long-range dependent by modeling token-level surprisal (bits) from PaLM/PaLM2 models over The Pile, estimating a Hölder exponent (~0.59), Hurst parameter (~0.70), Joseph exponent (~0.49), and showing the median Hurst parameter combined with BPB better predicts downstream performance.
Problem
Why next-token prediction yields strong LLM capabilities, formalized through the fractal and long-range dependence structure of language, and whether fractal parameters of model log-loss sequences predict downstream performance.
Approach
Treat per-token negative log-probabilities from an LLM as an increment process and estimate fractal parameters (self-similarity/Hölder exponent, Hurst parameter, fractal dimension, Joseph exponent) to quantify self-similarity and long-range dependence in language; combine median Hurst with BPB (as 1/BPB + H) to predict downstream accuracy.
Key results
- Median Hölder (self-similarity) exponent of language estimated at 0.59 ± 0.08 and median Hurst parameter at 0.70 ± 0.09 using PaLM2-L over The Pile.
- Median Hurst parameter correlates with BPB at absolute Pearson 0.83, and the combined 1/BPB + H predictor raises adjusted R^2 for downstream tasks (e.g., 0S BBH Direct 0.785 BPB to 0.883 combined).
- Context-length ablation with 1B T5.1.1 decoders (2K, 4K, 8K) showed no downstream improvement with longer training contexts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine median Hurst with reciprocal BPB as H_B = 1/BPB + H | Downstream performance prediction across the evaluated models · BBH and MMLU/GSM8K zero- and few-shot tasks | adjusted R²: 0.785 -> 0.883 (0S BBH Direct); 0.881 -> 0.979 (3S BBH CoT) | Helped |
| Use median Hurst instead of BPB alone | Downstream performance prediction across the evaluated models · BBH and MMLU/GSM8K zero- and few-shot tasks | adjusted R²: 0.653 -> 0.831 (0S MMLU); 0.654 -> 0.867 (8S GSM8K CoT) | Helped |
| Train T5.1.1 1B with longer context lengths (4K or 8K instead of 2K) (ablation) | T5.1.1 1B pretrained on SlimPajama-627B · Downstream benchmarks: 0S BBH Direct, 0S MMLU, 8S GSM8K CoT | raw accuracy: 1.81, 1.68, 1.76 (0S BBH Direct); 25.73, 26.04, 25.81 (0S MMLU); 1.06, 1.21, 1.74 (8S GSM8K CoT) | No clear effect |
| Combine BPB with self-similarity exponent S | Downstream performance prediction across the evaluated models · BBH, MMLU and GSM8K tasks | adjusted R²: 0.785 -> 0.761 (0S BBH Direct); 0.653 -> 0.614 (0S MMLU) | Hurt |
| Combine BPB with Joseph exponent J | Downstream performance prediction across the evaluated models · BBH, MMLU and GSM8K tasks | adjusted R²: 0.785 -> 0.794 (0S BBH Direct); 0.653 -> 0.614 (0S MMLU) | Mixed |
Surprising findings
- The median Hurst parameter, computed on zero-mean unit-variance token losses, is strongly correlated with BPB (|r|=0.83), suggesting second-order statistics of token losses predict mean loss.
- Training with longer contexts (2K, 4K, 8K) did not improve downstream performance in the tested setup, a negative result against the hypothesis.
Limitations
- Analysis is limited to English language text.
- Does not model semantic or lexical form of language.
- Main experiments rely on proprietary PaLM/PaLM2 models whose weights are not public.
Experimental setup
Datasets: The Pile validation split C4 SlimPajama-627B BIG-Bench Hard MMLU GSM8K
Benchmarks: BIG-Bench Hard MMLU GSM8K
Metrics: fractal parameters (Hölder exponent, Hurst, Joseph) bits-per-byte raw accuracy adjusted R^2 Pearson correlation
Baselines: bits-per-byte (BPB) self-similarity exponent Joseph exponent
Builds on: Hurst parameter R/S analysis bits-per-byte
Closest papers by meaning
- A Tale of Two Structures: Do LLMs Capture the Fractal Complexity of Language? (ICML 2025 · similarity 0.93)
- Correlation Dimension of Autoregressive Large Language Models (NeurIPS 2025 · similarity 0.89)
- Understanding LLM Behaviors via Compression: Data Generation, Knowledge Acquisition and Scaling Laws (NeurIPS 2025 · similarity 0.87)
- L$^2$M: Mutual Information Scaling Law for Long-Context Language Modeling (NeurIPS 2025 · similarity 0.86)
- Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training (NeurIPS 2025 · similarity 0.86)
- From Tokens to Words: On the Inner Lexicon of LLMs (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.