Research map / Large language models / Retrieval and knowledge
Understanding Data Temporality Impact on Large Language Models Pre-training
Romain Fabre, Hippolyte Pilchen, Franck SIGNE TALLA, Patrick Perez, Edouard Grave
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pre-train on chronologically ordered Common Crawl snapshots instead of a shuffled corpus | 6B decoder LLM · OLMES (general language understanding) | OLMES score | No clear effect |
| sequential ordering instead of shuffling, mid-training | 6B decoder LLM · OLMES learning dynamics | OLMES score | Hurt |
| sequential ordering instead of shuffling | 6B decoder LLM · KairosQA recent years (2023-2024) | accuracy and F1: relative F1 gain 16% for Seq vs negative gains for open-source shuffled models | Improved |
| sequential ordering instead of shuffling | 6B decoder LLM · KairosQA older years | accuracy | Hurt |
| do the cool-down of the 2021 sequential checkpoint on 2024 data instead of 2021 data (ablation) | Sequential 2021 checkpoint · OLMES | OLMES score (%): 71.7 -> 73.2 | Improved |
| vary shuffled pre-training length (token budget) (ablation) | shuffled baseline · KairosQA temporal alignment | accuracy by year | No clear effect |
| increase parameter count within a model family | open-source base models · KairosQA recent years | accuracy | No clear effect |
| increase number of cloze choices from 2 to 12 (ablation) | Sequential final checkpoint · KairosQA cloze 2024 | accuracy (%): 58 -> 22 | Hurt |
| query less popular subjects (ablation) | Sequential final checkpoint · KairosQA generative 2024 | F1: 0.4 -> 0.17 (top 10% to 30-40% bin) | Hurt |
| use TAQA benchmark to compare models | temporal evaluation · TAQA | F1: F1 from 1.5% to 10% | No clear effect |
Closest papers by meaning
- How Do Large Language Models Acquire Factual Knowledge During Pretraining? (NeurIPS 2024 · similarity 0.91)
- ChroKnowledge: Unveiling Chronological Knowledge of Language Models in Multiple Domains (ICLR 2025 · similarity 0.90)
- Arrows of Time for Large Language Models (ICML 2024 · similarity 0.88)
- Physics of Language Models: Part 3.1, Knowledge Storage and Extraction (ICML 2024 · similarity 0.88)
- Generalization v.s. Memorization: Tracing Language Models’ Capabilities Back to Pretraining Data (ICLR 2025 · similarity 0.88)
- What Do Learning Dynamics Reveal About Generalization in LLM Mathematical Reasoning? (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
