Research map / Architectures and efficiency / Attention and transformers
Technical Debt in In-Context Learning: Diminishing Efficiency in Long Context
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Benchmarks in-context learning of a GPT-2-style transformer on a hierarchical meta-ICL regression task against Bayes-optimal BMA and model-selection baselines (AIC, BIC, BMC), finding ICL is near-optimal in few-shot but its sample efficiency degrades in long context, and proves via information theory that this diminishing efficiency follows from a non-vanishing excess risk.
Problem
Whether in-context learning by transformers is an optimal learning algorithm compared to principled learners (e.g., Bayes-optimal estimator), and how its sample complexity scales with the number of demonstrations, especially in long context.
Approach
Benchmark ICL's sample complexity against principled learners using performance ratios and performance profiles (borrowed from optimization software benchmarking) in a meta-ICL hierarchical regression setup, and show with information-theoretic bounds that a non-vanishing excess risk in long context forces suboptimality that grows with performance requirements.
Key results
- ICL's mean performance ratio against BMA is at most 1.1 for performance quantiles up to 0.3, but grows to about 1.45 at quantile 0.99.
- For quantiles of 0.8 and above, ICL reaches only about 0.8 performance profile even at tau=3, while principled baselines (BIC, BMC) converge to near-perfect profiles.
- ICL's squared prediction distance to BMA plateaus after few demonstrations, whereas consistent estimators (BMC, BIC) converge to BMA; scaling pretraining prompt length or model size reduces excess risk magnitude but not the non-vanishing long-context shape.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ICL rather than BMA to reach low performance requirements (quantile ≤ 0.3) | ICL vs. BMA · Meta ICL scenarios | mean performance ratio: at most 1.1 | No clear effect |
| Use ICL rather than BMA for higher performance requirements, from quantile 0.3 to 0.99 | ICL vs. BMA · Meta ICL scenarios | mean performance ratio: around 1.1 at gQ=0.3; around 1.2 at gQ = 0.7; around 1.45 at gQ = 0.99 | Hurt |
| Use ICL rather than the best principled single-model baseline at low requirements (quantile ≤ 0.3) | ICL vs. AIC, BIC, and BMC · Meta ICL scenarios | performance profile at τ = 1: perfect | Helped |
| Use ICL rather than principled single-model baselines at high requirements (quantile ≥ 0.8) | ICL vs. AIC, BIC, and BMC · Meta ICL scenarios | performance profile at τ = 3: around 0.8 | Hurt |
| Increase pretraining prompt length | ICL · Meta ICL, long-context regimes | excess risk: 1.06, 0.58, and 0.34 | Helped |
| Increase model capacity | ICL · Meta ICL, long-context regimes | excess risk: 1.36, 1.19, 0.99, and 0.90 | Helped |
| Increase pretraining prompt length or model capacity | ICL · Meta ICL, long-context regimes | excess-risk curve shape | No clear effect |
| Replace the transformer with Mamba or LSTM | ICL sequence models · Meta ICL, long-context regimes | excess-risk curve shape | No clear effect |
Surprising findings
- ICL's sample-complexity efficiency deteriorates in many-shot regimes even though its error keeps decreasing with more demonstrations.
- Scaling model size or pretraining prompt length reduces excess risk magnitude but does not change the non-vanishing long-context shape.
Limitations
- Results are from stylized synthetic regression settings with a small GPT-2 model, not large real-world LLMs.
- Theoretical suboptimality bounds rely on Assumption of a non-vanishing excess risk lower bound after a reference length.
- The ICL objective differs from autoregressive LLM pretraining losses.
Experimental setup
Datasets: synthetic hierarchical meta-ICL regression prompts
Benchmarks: synthetic meta-ICL regression scenarios (M=10, sigma_y^2 in {0.003,0.03,0.3}, sigma_w^2 in {0.1,1,10})
Metrics: mean squared error performance ratio mean performance ratio performance profile squared prediction difference to BMA
Baselines: Bayesian model averaging (BMA) AIC BIC Bayesian model comparison (BMC) equal-weight ensemble of ridge estimators
Builds on: GPT-2 meta-learning in-context learning (Garg et al.) performance profiles (Dolan & More) Bayesian model averaging
Closest papers by meaning
- In-Context Learning through the Bayesian Prism (ICLR 2024 · similarity 0.93)
- Can In-context Learning Really Generalize to Out-of-distribution Tasks? (ICLR 2025 · similarity 0.92)
- Does learning the right latent variables necessarily improve in-context learning? (ICML 2025 · similarity 0.92)
- How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations (ICLR 2024 · similarity 0.91)
- Transformers are Minimax Optimal Nonparametric In-Context Learners (NeurIPS 2024 · similarity 0.91)
- Position: Do pretrained Transformers Learn In-Context by Gradient Descent? (ICML 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.