Research map / Large language models / Language, safety and interpretability
A Tale of Two Structures: Do LLMs Capture the Fractal Complexity of Language?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Investigates whether LLM-generated text replicates the fractal (Hölder self-similarity and Hurst long-range dependence) structure of human text across temperatures, prompts, and models, and releases the 240k-article GAGLE dataset.
Problem
Whether LLMs can replicate the fractal, self-similar and long-range-dependent information-theoretic complexity (bits per token) of natural language, and under which conditions (temperature, prompting, instruction tuning, model size) they fail.
Approach
Measures Hölder (self-similarity) and Hurst (long-range dependence) exponents of log-perplexity sequences for LLM-generated versus human text, varying decoding temperature, prompting strategy, instruction tuning and model size, and frames prompting as an interventional (do-) query to explain why contextual information can behave non-monotonically.
Key results
- Pretrained models at temperature 1.0 match human log-perplexity, while instruction-tuned and low-temperature outputs have lower log-perplexity than human text.
- Hurst exponent is a much stronger predictor of Gemini-rated text quality than log-perplexity or Hölder exponent; Hurst correlates with Hölder at Pearson 0.68.
- Prompting has the largest mutual information with fractal parameters (19.7% for Hurst, 8.4% for Hölder), and self-similarity shows a non-monotone double descent across prompt information density.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a larger pretrained model rather than a smaller pretrained model | Pretrained LLM generation at temperature β=1 · GAGLE | Closeness of Hölder and Hurst exponents to natural language | Helped |
| Use instruction-tuned generation rather than pretrained generation at low temperature | LLM generation with simple continuation · GAGLE and RAID | Hurst exponent | Hurt |
| Use instruction-tuned generation rather than pretrained generation | LLM generation with simple continuation · GAGLE and RAID | Hölder exponent | No clear effect |
| Provide a summary rather than unordered keywords in the prompt | Instruction-tuned LLM generation · GAGLE | Self-similarity closeness to natural language | Hurt |
| Provide ordered excerpts rather than a summary in the prompt | Instruction-tuned LLM generation · GAGLE | Self-similarity closeness to natural language | Helped |
| Ask the model to generate an outline before the article | Instruction-tuned LLM generation · GAGLE | Fractal-parameter closeness to natural language | Helped |
| Increase prompt information density | Instruction-tuned LLM generation · GAGLE | Self-similarity closeness to natural language | Mixed |
| Use lower decoding temperatures | LLM generation · GAGLE and RAID | Log-perplexity similarity to human text | Hurt |
| Use pretraining temperature β=1 rather than lower temperatures | Large pretrained LLM generation · GAGLE and RAID | Log-perplexity similarity to human text | Helped |
| Use prompting rather than other varied factors | Instruction-tuned LLM generation · GAGLE | Mutual information with Hölder exponent: Prompt: 8.4%; Dataset: 7.3%; Temp: 1.0%; Generating Model: 1.0%; Scoring Model: 0.2% | Helped |
| Use prompting rather than other varied factors | Instruction-tuned LLM generation · GAGLE | Mutual information with Hurst exponent: Prompt: 19.7%; Dataset: 7.2%; Generating Model: 4.8%; Temp: 4.3%; Scoring Model: 1.7% | Helped |
| Generate encyclopedic or legal text rather than other GAGLE domains | Instruction-tuned LLM generation · GAGLE | Closeness to natural language in log-perplexity and fractal parameters | Helped |
Surprising findings
- Providing unordered keywords yields better self-similarity match to human text than providing a summary, a double-descent pattern.
- Large pretrained models at temperature 1.0 produce log-perplexity similar to human text, undermining log-perplexity-based detection for them.
Limitations
- Fractal parameters require large corpora of long documents and are not reliable for individual documents, so they cannot detect single AI-generated texts.
- Only autoregressive models were studied; state space models were not examined.
- Estimates use truncated scales (max 320 tokens) and larger epsilon than prior work, making them less accurate.
Experimental setup
Datasets: WIKIPEDIA BIGPATENT NEWSROOM SCIENTIFIC BILLSUM RAID
Benchmarks: RAID
Metrics: Hölder exponent Hurst exponent log-perplexity mutual information classifier accuracy
Baselines: GLTR DetectGPT
Builds on: RAID benchmark chain-of-thought prompting GLTR
Closest papers by meaning
- Fractal Patterns May Illuminate the Success of Next-Token Prediction (NeurIPS 2024 · similarity 0.93)
- Correlation Dimension of Autoregressive Large Language Models (NeurIPS 2025 · similarity 0.91)
- Understanding LLM Behaviors via Compression: Data Generation, Knowledge Acquisition and Scaling Laws (NeurIPS 2025 · similarity 0.88)
- When LLMs Meet Cunning Texts: A Fallacy Understanding Benchmark for Large Language Models (NeurIPS 2024 · similarity 0.86)
- Arrows of Time for Large Language Models (ICML 2024 · similarity 0.86)
- Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.