Research map / Large language models / Language, safety and interpretability
From Unstructured Data to In-Context Learning: Exploring What Tasks Can Be Learned and When
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretically and empirically shows that in-context learning can arise from co-occurrence modeling with CBOW for word analogy tasks, while logic-reasoning ICL requires positional information and blocked nuisance structure, and identifies two data structures where ICL fails.
Problem
Existing theories of in-context learning assume structured training data resembling ICL tasks (e.g., x-y pairs), whereas LLMs are trained unsupervised on unstructured text; it is unclear what tasks can be learned in context from such data and which model components and data structures are needed.
Approach
Links ICL for word analogy to co-occurrence modeling with classical CBOW (no positional encoding or attention), shows positional information and blocked nuisance token structure are necessary for pattern-based logic reasoning ICL, and identifies training-data structures under which ICL provably fails.
Key results
- LLaMA 2 ICL accuracy on country-capital prompts is 0.58 for type A countries versus 0.96 for type B countries.
- A CBOW model achieves perfect ICL accuracy on single-relationship word analogy in many settings (e.g., (1/2,0,1/2) and (1/3,1/3,1/3) with both embedding sizes 10 and 100).
- A one-layer model without positional embeddings reaches only 0.30 accuracy on the first-token pattern task, while learned or sinusoidal positional embeddings reach 1.00.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use prompts whose capital is also the largest city rather than type A prompts | LLaMA 2 · Country-capital prompts | accuracy: 0.58 -> 0.96 | Helped |
| use prompts whose capital is also the largest city rather than type A prompts | LLaMA 2 · US state-capital prompts | accuracy: 0.69 -> 0.84 | Helped |
| use a consistent country-IOC position pattern rather than variable country-capital positions | 5-layer 2-head transformer · Synthetic corpus, ICL | accuracy: (0.00, 0.15, 0.34, 0.22, 0.07) vs (1.00, 0.77, 0.78, 0.97, 0.99) | Mixed |
| add learned positional embeddings instead of using sinusoidal embeddings (ablation) | single-pattern transformer · Single-pattern ICL, block-noisy, d_E = 100 | accuracy: 0.55 -> 1.00 | Helped |
| add sinusoidal positional embeddings rather than use learned embeddings (ablation) | single-pattern transformer · Single-pattern ICL, one-noisy, d_E = 10 | accuracy: 0.00 -> 0.10 | Helped |
| add learned positional embeddings instead of RoPE (ablation) | single-pattern transformer · Single-pattern ICL, block-noisy, d_E = 100 | accuracy: 0.00 -> 1.00 | Helped |
| use 5 layers instead of 1 layer (ablation) | dual-pattern transformer with learned positional embeddings · Dual-pattern ICL, clean, d_E = 100 | accuracy: (0.51, 0.49) -> (0.97, 0.98) | Helped |
| add learned positional embeddings instead of sinusoidal embeddings (ablation) | 5-layer dual-pattern transformer · Dual-pattern ICL, block-noisy, d_E = 100 | accuracy: (0.04, 0.07) -> (0.87, 0.70) | Helped |
| add positional embeddings (ablation) | 1-layer transformer · Position generalization, Both split | accuracy: 0.30 -> 1 | Helped |
| add positional embeddings (ablation) | 5-layer transformer · Position generalization, Both split | accuracy: 0.89 -> 1 | Helped |
| ensure every token appears as the first token in both training and test sets rather than splitting first tokens between sets (ablation) | transformer · Position generalization | accuracy: 0 -> 1 | Helped |
| include blocked nuisance tokens rather than one randomly inserted nuisance token (ablation) | position-aware autoregressive model · Single-pattern ICL | accuracy: 0.00 -> 1.00 | Helped |
Surprising findings
- ICL for word analogy can arise from co-occurrence modeling alone via CBOW, without positional information or attention.
- ICL fails for the repeated-meta-pattern task and for co-occurring pairs restricted to fixed positions, regardless of architecture.
Limitations
- Experiments are conducted at relatively small scale.
- Only specific ICL task types are studied.
- Pre-training data may not match grammatical natural language; real datasets are not used.
Experimental setup
Datasets: country-capital pairs from population data US state-capital pairs synthetic country-capital and country-IOC corpus synthetic pattern sequences (abacdc, abcadefd, etc.)
Metrics: ICL accuracy prediction accuracy
Baselines: LLaMA 2 five-layer autoregressive transformer one-layer transformer
Builds on: continuous bag of words (CBOW) induction heads RoPE
Closest papers by meaning
- In-Context Language Learning: Architectures and Algorithms (ICML 2024 · similarity 0.90)
- Revisiting In-context Learning Inference Circuit in Large Language Models (ICLR 2025 · similarity 0.90)
- In-Context Learning Learns Label Relationships but Is Not Conventional Learning (ICLR 2024 · similarity 0.89)
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.89)
- Capturing In-Context Learning Dynamics with Task Operators (NeurIPS 2026 · similarity 0.89)
- Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.