AI Paper Atlas

Research map / Large language models / Retrieval and knowledge

Understanding Data Temporality Impact on Large Language Models Pre-training

Romain Fabre, Hippolyte Pilchen, Franck SIGNE TALLA, Patrick Perez, Edouard Grave

ICML 2026 · Large language models › Retrieval and knowledge · approach: Knowledge editing in LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
pre-train on chronologically ordered Common Crawl snapshots instead of a shuffled corpus6B decoder LLM · OLMES (general language understanding)OLMES scoreNo clear effect
sequential ordering instead of shuffling, mid-training6B decoder LLM · OLMES learning dynamicsOLMES scoreHurt
sequential ordering instead of shuffling6B decoder LLM · KairosQA recent years (2023-2024)accuracy and F1: relative F1 gain 16% for Seq vs negative gains for open-source shuffled modelsImproved
sequential ordering instead of shuffling6B decoder LLM · KairosQA older yearsaccuracyHurt
do the cool-down of the 2021 sequential checkpoint on 2024 data instead of 2021 data (ablation)Sequential 2021 checkpoint · OLMESOLMES score (%): 71.7 -> 73.2Improved
vary shuffled pre-training length (token budget) (ablation)shuffled baseline · KairosQA temporal alignmentaccuracy by yearNo clear effect
increase parameter count within a model familyopen-source base models · KairosQA recent yearsaccuracyNo clear effect
increase number of cloze choices from 2 to 12 (ablation)Sequential final checkpoint · KairosQA cloze 2024accuracy (%): 58 -> 22Hurt
query less popular subjects (ablation)Sequential final checkpoint · KairosQA generative 2024F1: 0.4 -> 0.17 (top 10% to 30-40% bin)Hurt
use TAQA benchmark to compare modelstemporal evaluation · TAQAF1: F1 from 1.5% to 10%No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy