Research map / Architectures and efficiency / Attention and transformers
Pretrain–Test Task Alignment Governs Generalization in In-Context Learning
Mary Letey, Jacob A Zavatone-Veth, Yue M. Lu, Cengiz Pehlevan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the derived e_misalign alignment measure instead of simpler measures to predict ICL error | 2-layer softmax Transformer on linear regression ICL · Power-law and low-rank test covariances | Spearman correlation with ICL error: 0.99 (ours) vs 0.98, 0.96, 0.39 | Improved |
| increase pretraining task diversity kappa when train and test are poorly aligned (ablation) | Linear attention ICL model · Solvable model, fixed C_train, varied C_test | ICL test error | Mixed |
| pretrain on a more low-dimensional (higher spectral power) task covariance than the test one, at low task diversity (ablation) | Linear attention ICL model · Power-law task covariances | ICL test error | Improved |
| raise pretraining spectral power too far above the test power (ablation) | Linear attention ICL model · Power-law task covariances | ICL test error | Hurt |
| use CKA or population C_test C_train^-1 as alignment predictor instead of e_misalign | ICL error prediction · Linear attention, power-law C_train | monotonic relation to ICL error | Hurt |
Closest papers by meaning
- When can in-context learning generalize out of task distribution? (ICML 2025 · similarity 0.93)
- Differential learning kinetics govern the transition from memorization to generalization during in-context learning (ICLR 2025 · similarity 0.91)
- Can In-context Learning Really Generalize to Out-of-distribution Tasks? (ICLR 2025 · similarity 0.91)
- In-Context Learning through the Bayesian Prism (ICLR 2024 · similarity 0.90)
- Understanding the Generalization of In-Context Learning in Transformers: An Empirical Study (ICLR 2025 · similarity 0.90)
- Technical Debt in In-Context Learning: Diminishing Efficiency in Long Context (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
