Research map / Large language models / Language, safety and interpretability

Investigating the Overlooked Hessian Structure: From CNNs to LLMs

ICML 2025 · Large language models › Language, safety and interpretability · approach: Data and evaluation methodology

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that well-trained CNNs and LLMs have a power-law Hessian eigenvalue spectrum, gives a maximum-entropy interpretation, and finds the KS distance of this spectrum predicts LLM generalization better than sharpness measures.

Problem

Prior work described the Hessian spectrum of deep networks only roughly (a few large and many near-zero eigenvalues), with no statistical characterization of its shape or its link to generalization, especially for LLMs.

Approach

Identifies a power-law (Zipf-like) structure in the top Hessian eigenvalues of well-trained CNNs and LLMs, tests it with MLE and KS goodness-of-fit, interprets it via maximum entropy, and uses the KS distance as a generalization predictor.

Key results

• Power-law fits hold for well-trained LeNet, ResNet18, and GPT-2 models (KS distance below critical value) and fail for random initializations, e.g. GPT2-small pretrained on OpenWebText with dks 0.0418 vs critical 0.0430.
• On GPT2-nano pretraining, KS distance predicts test loss with Pearson correlation up to 0.923, while sharpness measures are nearly useless.
• On GPT2-small fine-tuning, KS distance correlates with test loss at Pearson 0.980, whereas the largest eigenvalue gives -0.714 and the trace gives 0.247.

Measured effects reported in the paper

Surprising findings

• Sharpness measures (largest eigenvalue, Hessian trace) that work well for CNNs largely fail to predict LLM generalization.
• Large-batch training can also produce power-law spectra, at the cost of more compute.

Limitations

• Authors could not test state-of-the-art LLMs due to the memory and compute cost of Hessian analysis.
• KS distance is not a robust generalization measure across all settings, per the authors' own conclusion.
• Analysis focuses on the top ~1000 eigenvalues, and the Hessian is computed on the last layer only for most experiments.

Experimental setup

Datasets: MNIST Fashion-MNIST CIFAR-10 CIFAR-100 Avila OpenWebText Shakespeare MathQA

Benchmarks: MNIST Fashion-MNIST CIFAR-10 CIFAR-100 OpenWebText Shakespeare MathQA

Metrics: Kolmogorov-Smirnov distance power exponent via MLE Pearson correlation with test loss largest Hessian eigenvalue Hessian trace

Baselines: sharpness measures (largest Hessian eigenvalue, Hessian trace)

Builds on: Davis-Kahan theorem Lanczos algorithm Powerlaw MLE (Clauset et al.)

Closest papers by meaning

• Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectra (ICLR 2025 · similarity 0.89)
• Neglected Hessian component explains mysteries in sharpness regularization (NeurIPS 2024 · similarity 0.88)
• Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Data (NeurIPS 2024 · similarity 0.88)
• Deeper or Wider: A Perspective from Optimal Generalization Error with Sobolev Loss (ICML 2024 · similarity 0.88)
• Do Neural Networks Need Gradient Descent to Generalize? A Theoretical Study (NeurIPS 2025 · similarity 0.87)
• Deep linear networks for regression are implicitly regularized towards flat minima (NeurIPS 2024 · similarity 0.87)