Research map / Architectures and efficiency / Attention and transformers
Understanding the Generalization of In-Context Learning in Transformers: An Empirical Study
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically studies how training-data coverage shapes in-context learning generalization across inter-problem, intra-problem, and intra-task dimensions, using function fitting with GPT-2/LLaMA-3 and tool-calling/translation experiments.
Problem
Understanding the boundaries of in-context learning generalization in transformers and how they depend on training data coverage, particularly under distribution shift.
Approach
A task-centric framework that decomposes training data into problems with nested tasks and tests three generalization types; finds transformers fail inter-problem generalization but succeed at intra-problem and intra-task generalization, and that training on more diverse mixed tasks improves ICL on unseen and even simple tasks.
Key results
- On convex-combination function fitting, the ComFuncLearner trained on one combination markedly lowers squared error on unseen combinations versus the Baseline (e.g., mean SE over 21-30 points drops from 0.0213 to 0.00294 for products with 4 training combinations).
- On ToolBench, CFL+ICL_m reaches 32.0 average on multi-API tasks versus ToolLLaMA's 29.0, and CFL improves over Baseline on single-API tasks too.
- On WMT14 mixed-language input, CFL_mix achieves 35.74 BLEU (mix2en) vs 33.31 for the best Baseline; ICL gains are larger on hard WMT500 samples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on multi-API examples rather than only single-API examples | LLaMA-2-7B with multi-API ICL · ToolBench I2/I3 | accuracy: 26.3 -> 32.0 | Helped |
| Train on multi-API examples rather than only single-API examples | LLaMA-2-7B with single-API ICL · ToolBench I1 | accuracy: 29.8 -> 30.9 | Helped |
| Add in-context examples to mixed-language translation | CFL_e2d · WMT14 mix2en | BLEU: 37.02 -> 37.69 | Helped |
| Add in-context examples to English-to-German baseline translation | BL_e2d · WMT14 en2de | BLEU: 24.81 -> 25.78 | Helped |
| Add in-context examples to CFL translation | CFL_e2d · WMT14 de2en | BLEU: 24.36 -> 24.36 | No clear effect |
| Add a convex-combination pattern to training | GPT-2-Standard function fitting · Convex combinations of base functions | squared error: 5.58e-2 -> 1.88e-3 | Helped |
| Increase product-combination training patterns from one to four (ablation) | GPT-2-Standard function fitting · Product combinations, points 21-30 | squared error: 6.83e-3 -> 2.94e-3 | Helped |
| Increase compositional-combination training patterns from one to four (ablation) | GPT-2-Standard function fitting · Compositional combinations, points 21-30 | squared error: 2.48e-2 -> 1.80e-2 | Helped |
| Train with a convex combination instead of an additional base function | GPT-2-Standard function fitting · Combination of sin(x) and cos(x) | squared error: 1.86e-3 -> 3.13e-3 | No clear effect |
| Expose the pretrained model to function-composition data during finetuning | LLaMA-3 8B · Function-fitting combinations | generalization | Helped |
| Train on mixed translation problems rather than only one direction | Qwen2-1.5B architecture · WMT14 mixed-language input | BLEU: 33.57 -> 37.69 | Helped |
| Train on mixed translation problems rather than only one direction | Qwen2-1.5B architecture · WMT14 en2de | BLEU: 24.81 -> 21.83 | Hurt |
Surprising findings
- Pretrained LLaMA-3 8B, like a randomly initialized GPT-2, fails to inter-problem generalize via ICL despite extensive pretraining.
- Training on a wider variety of mixed tasks improved ICL even on known simple tasks in some settings.
Limitations
- Experiments use small or specific settings (9.5M GPT-2, sinusoidal function families), limiting generality.
- Translation models are small randomly initialized models pretrained on limited data (2M sentences), and GPT-4o judging is a proxy evaluation.
Experimental setup
Datasets: ToolBench CC100 WMT14 WMT19 WizardLM_evol_instruct_70k
Benchmarks: ToolBench WMT14 de2en WMT19 WMT500
Metrics: squared error API-calling accuracy BLEU BERTScore LLM-as-a-judge score
Baselines: ToolLLaMA Baseline (single-task trained model)
Builds on: ToolLLaMA ReAct Qwen2
Closest papers by meaning
- How Do Nonlinear Transformers Learn and Generalize in In-Context Learning? (ICML 2024 · similarity 0.92)
- Transformers are Minimax Optimal Nonparametric In-Context Learners (NeurIPS 2024 · similarity 0.91)
- Towards Understanding How Transformers Learn In-context Through a Representation Learning Lens (NeurIPS 2024 · similarity 0.90)
- How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations (ICLR 2024 · similarity 0.90)
- In-Context Learning with Representations: Contextual Generalization of Trained Transformers (NeurIPS 2024 · similarity 0.90)
- Technical Debt in In-Context Learning: Diminishing Efficiency in Long Context (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.