Research map / Large language models / Agents, code and math

Neuron-based Multifractal Analysis of Neuron Interaction Dynamics in Large Models

ICLR 2025 · Large language models › Agents, code and math · approach: Mathematical and structural reasoning benchmarks · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Represents large models as weighted neuron interaction networks and applies neuron-based multifractal analysis to define a structure-based "degree of emergence" metric tracked over Pythia training checkpoints.

Problem

Emergent abilities in large models are studied mainly via size-performance relationships and black-box output evaluation, lacking systematic quantitative analysis of the internal structural dynamics driving emergence.

Approach

Abstracts a neural network into a Neuron Interaction Network (edge weights |w|^-1 with shortest-path distances), applies box-covering multifractal analysis per neuron to get regularity (alpha_0) and heterogeneity (spectrum width w), and combines them into a degree-of-emergence metric E = (w(t)/w(0)) log(alpha_0(0)/alpha_0(t)).

Key results

• The proposed degree-of-emergence metric E tracks benchmark accuracy trends (e.g., LAMBADA, PIQA) across Pythia training checkpoints for 160M, 1.4B and 2.8B models.
• At 53,000 epochs, the degree of emergence increases logarithmically with model size across Pythia models.
• Spectrum width w broadens over training for models from 14M to 410M and stabilizes after about 35,000 epochs for models above 1B.

Measured effects reported in the paper

Surprising findings

• For the 14M model, alpha_0 increases rather than decreases over training, so regularity does not increase for the smallest model.
• Benchmarks such as WSC and LogiQA show significant performance oscillations, which the authors argue limits their usefulness for assessing capabilities.

Limitations

• Validation is restricted to a few benchmarks; the authors acknowledge a more comprehensive evaluation is needed.
• Emergence metric is undefined/not computed for the smallest models (14M, 31M) because self-organization is not observed there.
• Correlation with performance is shown qualitatively on a limited set of models and checkpoints rather than established causally.

Experimental setup

Datasets: LAMBADA PIQA SciQ ARC-easy WSC LogiQA

Benchmarks: LAMBADA PIQA SciQ ARC-easy ARC Challenge WSC LogiQA

Metrics: accuracy degree of emergence E alpha_0 spectrum width w

Baselines: PASSUNTIL pre-training loss emergence analysis

Builds on: box-covering method box-growing method Watts-Strogatz network model

Closest papers by meaning

• Scaling Large Language Model-based Multi-Agent Collaboration (ICLR 2025 · similarity 0.87)
• Neuron Populations Exhibit Divergent Selectivity with Scale (NeurIPS 2026 · similarity 0.86)
• Large Language Models as Model Organisms for Human Associative Learning (NeurIPS 2025 · similarity 0.86)
• Scaling Laws and Tradeoffs in Recurrent Networks of Expressive Neurons (NeurIPS 2026 · similarity 0.86)
• BLEND: Behavior-guided Neural Population Dynamics Modeling via Privileged Knowledge Distillation (ICLR 2025 · similarity 0.86)
• The Emergence of Abstract Thought in Large Language Models Beyond Any Language (NeurIPS 2025 · similarity 0.86)