Research map / Large language models / Language, safety and interpretability

Position: Understanding LLMs Requires More Than Statistical Generalization

ICML 2024 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Position paper arguing LLMs should be studied in the saturation regime of test loss, using non-identifiability of autoregressive models to show that rule extrapolation, in-context learning, and fine-tunability are not explained by statistical generalization.

Problem

Statistical generalization (low train and test loss) does not explain desirable LLM properties such as zero-shot OOD rule extrapolation, in-context learning, and data-efficient fine-tunability, since autoregressive models with near-identical likelihood can behave very differently.

Approach

Argues for a shift from the interpolation regime to a "saturation regime" where test loss is near-minimal, and formalizes this via non-identifiability of AR probabilistic models (functional, epsilon-, and parameter non-identifiability), with three case studies attributing the behaviours to inductive biases.

Key results

• Transformer trained on a^nb^n PCFG extrapolates rule R1 on OOD prompts in 43.7% of cases while the loss is agnostic to this.
• Adversarial and oracle variants reach the same minimal test loss but show widely varying rule extrapolation performance.
• A proposition shows that for any epsilon>0, a distribution within epsilon KL of a mixture-of-HMMs pre-training distribution can fail to perform in-context learning, even with arbitrarily long prompts.

Measured effects reported in the paper

Surprising findings

• Models matching the pre-training distribution within epsilon KL can fail to exhibit in-context learning, even with increasing prompt size.
• Models with identical test loss show widely varying zero-shot rule extrapolation, so loss alone does not determine OOD behaviour.

Limitations

• Empirical evidence is limited to a toy a^nb^n PCFG setting and small Transformers.
• Theoretical ICL result relies on prior assumptions from the mixture-of-HMMs framework and omitted assumptions from cited work.
• Research directions are largely open questions rather than validated methods.

Experimental setup

Datasets: a^nb^n probabilistic context-free grammar samples

Benchmarks: a^nb^n zero-shot rule extrapolation on OOD prompts of length 8

Metrics: proportion of rule R1-consistent completions test loss (KL divergence)

Builds on: PAC-Bayes Transformers in-context learning as implicit gradient descent Bayesian in-context learning with HMM mixtures (Xie et al.) grokking

Closest papers by meaning

• On the Identifiability of Hybrid Deep Generative Models: Meta-Learning as a Solution (NeurIPS 2024 · similarity 0.87)
• Identifiability of deep generative models without auxiliary information (NeurIPS 2022 · similarity 0.87)
• Position: Deep Learning is Not So Mysterious or Different (ICML 2025 · similarity 0.86)
• Interpretability Illusions in the Generalization of Simplified Models (ICML 2024 · similarity 0.85)
• Understanding Domain Generalization: A Noise Robustness Perspective (ICLR 2024 · similarity 0.85)
• Simplicity Bias in Overparameterized Machine Learning (AAAI 2024 · similarity 0.85)