Research map / Large language models / Language, safety and interpretability
Towards Auto-Regressive Next-Token Prediction: In-context Learning Emerges from Generalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a PAC-Bayesian theory of in-context learning under auto-regressive next-token prediction, with two-level (sequence and topic) expectations, arguing ICL emerges from generalization of sequences and topics, and checks it on linear dynamic systems, GINC, and real-world GPT-2 experiments.
Problem
Existing ICL theory assumes i.i.d. prompt input-label pairs and explains what ICL computes (implicit optimization) rather than how ICL emerges from pre-training under realistic token-dependent auto-regressive next-token prediction.
Approach
Models pre-training and ICL under AR-NTP with a layer-wise topic/sequence structure and a two-level expectation (over sequences and topics), then derives data-dependent, topic-dependent, and optimization-dependent PAC-Bayesian bounds on the population loss, using ghost sequences to handle token dependency.
Key results
- On GINC, increasing sequences per topic N, sequence length T, and topics K each improves ICL generalization, consistent with the O(1/sqrt(KNT)) and O(1/sqrt(KT_p)) bound terms.
- Prior model initialization with GPT2-small warmup reduces GPT2-large training time for 30,000 steps from about 7 hours (random init, four A100s) to about 4 hours plus 0.5 hours for the small model.
- On linear dynamic systems, ICL loss falls sharply as topic count K grows past certain thresholds, which the authors read as emergent behavior.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase pre-training sequences per topic | GPT-2 on GINC · GINC | generalization performance | Helped |
| Increase pre-training sequence length | GPT-2 on GINC · GINC | generalization performance | Helped |
| Increase the number of pre-training topics | GPT-2 on GINC · GINC | generalization performance | Helped |
| Increase ICL prompt length | GPT-2 on GINC · GINC | generalization performance | Helped |
| Increase vocabulary size | GPT-2 on GINC · GINC | ICL prediction accuracy | Helped |
| Use random transitions in pre-training data | GPT-2 on GINC · GINC | ICL ability | Hurt |
| Increase the number of pre-training topics | GPT-2 on a linear dynamic system · Linear dynamic system | ICL loss | Helped |
| Increase the number of sequences per topic | GPT-2 on a linear dynamic system · Linear dynamic system | ICL loss | Helped |
| Increase pre-training sequence length | GPT-2 on a linear dynamic system · Linear dynamic system | ICL loss | Helped |
| Train with a subset of topics to obtain the topic-dependent prior instead of using the complete topic set | GPT-2 on a linear dynamic system · Linear dynamic system | steady-state training loss | No clear effect |
| Increase the number of training sequences | GPT2-large on real-world language datasets · Real-world language datasets | convergence speed | Helped |
| Increase the number of pre-training sequence tokens | GPT2-large on real-world language datasets · Real-world language datasets | generalization performance | Helped |
Surprising findings
- When pre-training data contains random transitions, the model observes all token transitions yet ICL fails, which the authors attribute to data distributions not matching the topic.
- A larger vocabulary size leads to higher ICL prediction accuracy on GINC.
Limitations
- Assumptions such as bounded log-likelihood ratio and bounded gradients are strong and only loosely justified (e.g., scaling-law estimate for S).
- Experiments are on small synthetic data (GINC, linear dynamics) and only a limited real-world setup, so empirical validation of the bounds is qualitative.
- The bounds are not shown to be tight, and the ICL-versus-meta-learning distinction is argued informally.
Experimental setup
Datasets: GINC linear dynamic system synthetic data 20 real-world language pre-training datasets (sentiment analysis, question answering, reasoning)
Benchmarks: GINC linear dynamic system real-world language datasets
Metrics: ICL loss overall prediction loss ICL accuracy training time
Baselines: algorithm stability meta-learning bound
Builds on: PAC-Bayesian generalization bounds GINC in-context learning dataset data-dependent prior
Closest papers by meaning
- In-Context Language Learning: Architectures and Algorithms (ICML 2024 · similarity 0.90)
- In-Context Learning Learns Label Relationships but Is Not Conventional Learning (ICLR 2024 · similarity 0.90)
- Position: Do pretrained Transformers Learn In-Context by Gradient Descent? (ICML 2024 · similarity 0.89)
- In-Context Learning through the Bayesian Prism (ICLR 2024 · similarity 0.89)
- Revisiting In-context Learning Inference Circuit in Large Language Models (ICLR 2025 · similarity 0.89)
- In-Context Learning with Representations: Contextual Generalization of Trained Transformers (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.