Research map / Large language models / Language, safety and interpretability
Correlation Dimension of Autoregressive Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes correlation dimension computed on next-token log-probability vectors as a fractal-geometric, inference-time metric for LLM perceived complexity, showing it tracks pretraining phases, hallucination, and degeneration.
Problem
Perplexity and standard local or global text metrics fail to capture long-range structural complexity of LLM-generated text, leaving behaviors such as repetition, incoherence, blandness, and hallucination poorly characterized even at low perplexity.
Approach
Treat the sequence of next-token log-probability vectors as a trajectory and compute its correlation dimension (scaling exponent of the correlation integral over recurrences), which is computable from logits already produced at inference, with fused CUDA kernels and vocabulary projection for efficiency.
Key results
- Correlation dimension of natural text converges to about 6.5 across pretrained LLMs while shuffled text exceeds 10 and repetitive text is below 2.
- Degenerate texts (repetitive, incoherent, bland) have significantly lower CorrDim than normal responses on Falcon3-10B (e.g., repetitive 3.80 vs normal 5.04, p=9.5E-7).
- CorrDim on random-name completion stress test correlates with HelloEval scores (Spearman rho=0.952); quantized AWQ/GPTQ models change CorrDim by mean absolute 0.14.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| generate repetitive, incoherent, or bland responses instead of normal responses | texts evaluated with Falcon3-10B · Controlled degeneration responses | correlation dimension: 5.04 -> 3.80; 5.04 -> 3.96; 5.04 -> 4.51 | Hurt |
| use perplexity to assess degeneration across types | texts evaluated with Falcon3-10B · Controlled degeneration responses | perplexity: 10.79 -> 1.25; 10.79 -> 13.24; 10.79 -> 4.24 | Mixed |
| use a model with greater robustness on random-name completion | language models · Random-name completion, input length n=512 | correlation dimension: 3.54 -> 8.89 | Helped |
| successfully recall names rather than hallucinate them | models completing SEP process-theism · SEP process-theism | correlation dimension: 3.56 -> 6.68; 3.56 -> 8.49 | Helped |
| limit context length to 32 tokens rather than use very short context | pretrained language models · SEP dataset | correlation dimension: approximately 3 to about 8 | Helped |
| extend context beyond 32 tokens | pretrained language models · SEP dataset | correlation dimension: about 8 to around 6.5 | Mixed |
| use higher-order delayed embeddings instead of single-step log-probability vectors (ablation) | correlation-dimension estimation · Text sequences | correlation dimension | No clear effect |
| reduce vocabulary dimension from 151643 to 10000 (ablation) | correlation-dimension estimation with Qwen2.5-7B · SEP newton-philosophy | correlation dimension: 5.92 -> 5.81 | No clear effect |
| quantize the model with AWQ (ablation) | Qwen2.5-32B-Instruct · SEP dataset | mean correlation dimension: 5.86 -> 5.90; mean absolute change 0.14 | No clear effect |
| quantize the model with GPTQ (ablation) | Qwen2.5-32B-Instruct · SEP dataset | mean correlation dimension: 5.86 -> 5.93; mean absolute change 0.14 | No clear effect |
| fuse pairwise distance calculation with correlation counting in one CUDA kernel | correlation-dimension computation · Sequence length 50,000; vector dimension 10,000 | speedup: 0.92x -> 1.83x | Helped |
Surprising findings
- Falcon3-7B and -10B recall obscure names with high CorrDim while Qwen2.5 models of similar or larger size hallucinate with low CorrDim, so model size does not determine the signal.
- Smaller Pythia models (14M, 160M) show a sudden rise in correlation dimension late in training that coincides with a drop in in-context learning accuracy.
Limitations
- Requires access to full logits, so it cannot be applied to closed API models such as GPT-4.
- Finite sequences do not guarantee a power-law correlation integral; the slope estimate depends on chosen clipping range settings.
- Correlation dimension of a finite sequence is an empirical quantity without formal estimation guarantees.
Experimental setup
Datasets: Stanford Encyclopedia of Philosophy (SEP) 60 longest articles Project Gutenberg books in 8 languages Python, Java, C standard library source files Random English names list GPT-4o-generated degenerate responses to 20 questions
Benchmarks: SEP dataset HelloEval
Metrics: correlation dimension perplexity Spearman correlation Wilcoxon signed-rank p-value Rep-2
Baselines: Perplexity Zipf coefficient Heap coefficient Rep-N Distinct-N Self-BLEU BERTScore MAUVE HelloEval
Builds on: Takens embedding theorem FlashAttention GPTQ AWQ
Closest papers by meaning
- A Tale of Two Structures: Do LLMs Capture the Fractal Complexity of Language? (ICML 2025 · similarity 0.91)
- Fractal Patterns May Illuminate the Success of Next-Token Prediction (NeurIPS 2024 · similarity 0.89)
- Characterizing Truthfulness in Large Language Model Generations with Local Intrinsic Dimension (ICML 2024 · similarity 0.89)
- Understanding LLM Behaviors via Compression: Data Generation, Knowledge Acquisition and Scaling Laws (NeurIPS 2025 · similarity 0.89)
- Arrows of Time for Large Language Models (ICML 2024 · similarity 0.89)
- Lines of Thought in Large Language Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.