Research map / Learning theory and algorithms / Kernels and regression theory

An Information-Theoretic Analysis of In-Context Learning

ICML 2024 · Learning theory and algorithms › Kernels and regression theory · approach: Statistical perspectives on generalization · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces information-theoretic tools that decompose Bayes-optimal meta-learning log-loss over sequences into irreducible, meta-estimation, and intra-document estimation error, and applies them to bound in-context learning error for a sparse mixture of transformers, with no mixing-time assumptions.

Problem

Existing theory of in-context learning and meta-learning from sequences relies on contrived mixing-time or stability assumptions and often fails to show error decaying with both sequence count and sequence length.

Approach

Decomposes the Bayes-optimal average cumulative log-loss exactly via the chain rule of mutual information into irreducible error plus meta and intra-document estimation terms, and uses rate-distortion theory with a KL distortion to bound the estimation terms for transformer and mixture-of-transformer environments.

Key results

• Exact decomposition: average log-loss of Bayes-optimal predictor equals irreducible error plus meta estimation error I(H_{M,T};psi)/(MT) plus intra-document error I(D_m;theta_m|psi)/T.
• For a sparse mixture of transformers with Dirichlet(R/N) prior, the meta estimation error scales linearly in R and logarithmically in N and M, with the intra-document term at most log(N)/T.
• Transformer estimation error bound decays linearly in T and scales with (d^2+r^2)L^2 log(K) terms, without mixing-time assumptions.

Surprising findings

• The in-context error is bounded by log(N)/tau, suggesting ICL only needs to identify which mixture component generated the sequence, consistent with reports that scrambled labels barely hurt performance.

Limitations

• Results are for exact Bayesian inference; the main text does not analyze practical learners, with suboptimal algorithms only in appendix.
• No empirical validation of the bounds on real LLMs or trained transformers.
• Bounds on transformer environment rely on Gaussian weight assumptions and a fixed-embedding, clipped transformer model.

Experimental setup

Metrics: expected log-loss estimation error bound

Builds on: Bayesian posterior prediction log-loss rate-distortion theory Xie et al. 2021 implicit Bayesian inference in ICL

Closest papers by meaning

• In-Context Learning through the Bayesian Prism (ICLR 2024 · similarity 0.88)
• In-Context Learning Strategies Emerge Rationally (NeurIPS 2025 · similarity 0.87)
• Unsupervised Meta-Learning via In-Context Learning (ICLR 2025 · similarity 0.87)
• In-context Learning in Presence of Spurious Correlations (NeurIPS 2026 · similarity 0.86)
• Technical Debt in In-Context Learning: Diminishing Efficiency in Long Context (NeurIPS 2025 · similarity 0.86)
• In-Context Learning and Occam's Razor (ICML 2025 · similarity 0.86)