Research map / Large language models / Reasoning and chain of thought

Universal One-third Time Scaling in Learning Peaked Distributions

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that softmax and cross-entropy alone produce power-law loss decay with training time (exponent 1/3) when learning peaked distributions, using a teacher-student toy model with low-temperature expansion and validates on Pythia and OLMo checkpoints.

Problem

Origin of slow power-law loss convergence in LLM training with time/data; existing theories attribute it to power-law data structure, but the role of the softmax and cross-entropy nonlinearities is unexplored.

Approach

Low-temperature expansion of the free energy of a Boltzmann (softmax) distribution gives internal energy U(β) = -c0 + c2 β^-2, so under gradient flow of an aligned student the inverse temperature grows as τ^(1/3) and loss decays as τ^(-1/3), independent of data power laws.

Key results

• Toy model with aligned student gives loss L ~ tau^(-1/3) in the intermediate low-temperature regime; fitted student inverse-temperature exponent 0.38 under optimal learning rate.
• Pythia checkpoints' loss minus constant fits a power law in dynamic time with fitted exponents very close to 1/3 across model sizes, with curves collapsing across sizes.
• Estimated next-token entropy bound of 1.69 implies equivalent i.i.d. Gaussian logit std 6.87, above c0≈4.55, placing LLMs in the low-temperature regime.

Surprising findings

• Loss exponent 1/3 arises from softmax and cross-entropy alone, without power-law structure in data.
• Under weight decay, the 1/3 loss scaling persists even while logit norm stops growing, attributed to parameter rotation.

Limitations

• Analysis uses gradient flow; effects of gradient noise and large learning rates are not analyzed.
• Aligned student ansatz is heuristic and cannot explain the rotation-driven 1/3 scaling seen with weight decay.
• LLM validation relies on fitted power laws and an equivalent Gaussian-logit approximation of entropy, which is not directly comparable.

Experimental setup

Datasets: FineWeb

Benchmarks: FineWeb

Metrics: test loss fitted power-law exponent logit standard deviation

Baselines: Chinchilla scaling laws

Builds on: Chinchilla scaling laws Adam Pythia

Closest papers by meaning

• Scaling Law with Learning Rate Annealing (NeurIPS 2025 · similarity 0.92)
• Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolation (NeurIPS 2026 · similarity 0.90)
• The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws (ICLR 2025 · similarity 0.90)
• A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.89)
• Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedules (NeurIPS 2025 · similarity 0.89)
• Resolving Discrepancies in Compute-Optimal Scaling of Language Models (NeurIPS 2024 · similarity 0.89)