Research map / Large language models / Language, safety and interpretability
The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Design
Yoav Levine, Noam Wies, Daniel Jannai, Dan Navon, Yedid Hoshen, Amnon Shashua
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Append retrieved Wikipedia neighbors in-context during TAPT rather than using regular TAPT | RoBERTa-base · SentEval sentence similarity (Avg.) | sentence similarity score: 59.8 -> 61.0 | Improved |
| Use retrieved neighbors in separate in-batch examples rather than regular TAPT (ablation) | RoBERTa-base · SentEval sentence similarity (Avg.) | sentence similarity score: 59.8 -> 58.8 | Hurt |
| Use random Wikipedia sentences in separate in-batch examples rather than regular TAPT (ablation) | RoBERTa-base · SentEval sentence similarity (Avg.) | sentence similarity score: 59.8 -> 58.4 | Hurt |
| Append random Wikipedia sentences in-context rather than using regular TAPT (ablation) | RoBERTa-base · SentEval sentence similarity (Avg.) | sentence similarity score: 59.8 -> 60.0 | No clear effect |
| Use retrieved neighbors in-context rather than retrieved neighbors in separate in-batch examples (ablation) | kNN-TAPT · SentEval sentence similarity (Avg.) | sentence similarity score: 58.8 -> 61.0 | Improved |
| Use retrieved neighbors rather than random Wikipedia sentences in-context (ablation) | kNN-TAPT · SentEval sentence similarity (Avg.) | sentence similarity score: 60.0 -> 61.0 | Improved |
| Add kNN-Pretraining after regular Wikipedia pretraining | GPT-2-medium · Natural Questions open-domain QA, zero-shot closed-book | correct answers: less than 50 -> roughly 250 | Improved |
Closest papers by meaning
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.88)
- In-Context Language Learning: Architectures and Algorithms (ICML 2024 · similarity 0.88)
- Once Read is Enough: Domain-specific Pretraining-free Language Models with Cluster-guided Sparse Experts for Long-tail Domain Knowledge (NeurIPS 2024 · similarity 0.88)
- In-Context Pretraining: Language Modeling Beyond Document Boundaries (ICLR 2024 · similarity 0.87)
- An Analysis for Reasoning Bias of Language Models with Small Initialization (ICML 2025 · similarity 0.87)
- Understanding In-Context Learning from Repetitions (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
