Research map / Large language models / Language, safety and interpretability
In-Context Pretraining: Language Modeling Beyond Document Boundaries
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes In-Context Pretraining, which reorders pretraining documents so each context window holds related documents (via contriever nearest-neighbor search and a max-TSP-style graph traversal), and pretrains 0.3B–7B LLaMA-style models from scratch on 306B tokens.
Problem
Standard LM pretraining concatenates random short documents into context windows, so prior documents give no signal for predicting the next document; the goal is to make pretraining contexts contain related documents at web scale without repeating data.
Approach
Keep the standard pretraining pipeline but reorder documents: link each document to its nearest neighbors via dense retrieval and FAISS, then build contexts by a greedy maximum-weight path traversal that visits each document exactly once, instead of kNN grouping that duplicates documents.
Key results
- In-context pretraining improves average in-context classification accuracy over standard pretraining by about 8% (71.3 vs 66.0 average across seven datasets with 32 demonstrations).
- Average reading comprehension accuracy rises from 37.6 (standard) to 43.2 across six datasets with 2-shot prompting, with HotpotQA going from 10.5 to 21.9.
- Open-book NQ exact match increases from 28.5 (standard) to 32.2 with Wikipedia retrieval, while kNN pretraining degrades to 20.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pretrain on sequences of related documents instead of randomly ordered documents | ICLM · Text classification (seven datasets) | average accuracy: 66.0 -> 71.3 | Helped |
| Pretrain on sequences of related documents instead of randomly ordered documents | ICLM · Reading comprehension (six datasets) | average score: 37.6 -> 43.2 | Helped |
| Pretrain on sequences of related documents instead of randomly ordered documents | ICLM · NQ and TriviaQA open-book | exact match: +9% | Helped |
| Pretrain on sequences of related documents instead of randomly ordered documents | ICLM · NQ and TriviaQA closed-book | exact match | Mixed |
| Pretrain on sequences of related documents instead of randomly ordered documents | ICLM · NQ-Swap and MemoTrap | exact match | Helped |
| Pretrain on sequences of related documents instead of randomly ordered documents | ICLM · SCROLL long-context tasks | average score: 32.5 -> 34.1 | Helped |
| Use kNN document grouping instead of random document ordering | Standard pretraining · Wikipedia, Arxiv, and Books language modeling | perplexity | No clear effect |
| Increase document relevance from random to clustering to nearest-neighbor linking (ablation) | ICLM · Wikipedia | perplexity | Helped |
| Add semantic deduplication of near-duplicate documents (ablation) | ICLM · Wikipedia | perplexity | Helped |
| Increase the number of in-context demonstration examples (ablation) | 7B ICLM · Text classification | accuracy | Helped |
Surprising findings
- Naive kNN grouping of related documents performs worse than standard pretraining on several tasks, attributed to repeated documents causing overfitting.
- Removing semantic deduplication noticeably increases perplexity, since near-duplicate documents in the same context let models copy from prior text.
Limitations
- Closed-book retrieval performance is comparable or slightly worse than standard, and gains on long-context fine-tuning (SCROLL) are smaller.
- Experiments use a single corpus (CommonCrawl) and English only, with a sample of 235M documents due to resource constraints.
Experimental setup
Datasets: English CommonCrawl (235M-document sample, ~306B tokens) SCROLL (finetuning for long-context evaluation)
Benchmarks: Wikipedia perplexity Arxiv perplexity Books perplexity SST2 Amazon Yelp AGNews DBpedia Hate Speech Offensive RACE BoolQ
Metrics: perplexity accuracy exact match F1 ROUGE-1
Baselines: Standard pretraining kNN pretraining clustering-based grouping
Builds on: LLaMA Contriever FAISS retrieval-augmented language model pretraining
Closest papers by meaning
- Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scale (NeurIPS 2026 · similarity 0.88)
- NExtLong: Toward Effective Long-Context Training without Long Documents (ICML 2025 · similarity 0.88)
- Data Engineering for Scaling Language Models to 128K Context (ICML 2024 · similarity 0.88)
- Doc-to-LoRA: Learning to Instantly Internalize Contexts (ICML 2026 · similarity 0.87)
- Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum (NeurIPS 2024 · similarity 0.87)
- Short-Context Dominance: How Much Local Context Natural Language Actually Needs? (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.