Research map / Large language models / Retrieval and knowledge
Mind the Gap: Assessing Temporal Generalization in Neural Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train on data overlapping the test period (CONTROL) instead of time-stratified past data | Transformer-XL 287M · CUSTOMNEWS | perplexity: 21.33 -> 18.38 | Helped |
| train on data overlapping the test period (CONTROL) instead of time-stratified past data | Transformer-XL 287M · WMT | perplexity: 22.45 -> 21.11 | Helped |
| train on data overlapping the test period (CONTROL) instead of time-stratified past data | Transformer-XL 287M · arXiv | perplexity: 23.07 -> 21.38 | Helped |
| increase model size from 287M to 448M parameters | time-stratified Transformer-XL · WMT and CUSTOMNEWS temporal degradation | relative perplexity slope over time: 5.5% lower absolute perplexity | No clear effect |
| add dynamic evaluation (update all parameters on test stream) | time-stratified Transformer-XL · WMT | perplexity: 22.45 -> 22.17 | Helped |
| add dynamic evaluation (update all parameters on test stream) | time-stratified Transformer-XL · arXiv | perplexity: 23.07 -> 20.98 | Helped |
| add dynamic evaluation (update all parameters) | time-stratified Transformer-XL · WMT new/shifting words | perplexity: 109.73 -> 66.26 | Helped |
| update only bias terms instead of all parameters in dynamic evaluation (ablation) | time-stratified Transformer-XL · CUSTOMNEWS | perplexity: 20.72 -> 20.96 | No clear effect |
| update only embeddings instead of all parameters in dynamic evaluation (ablation) | time-stratified Transformer-XL · arXiv | perplexity: 20.98 -> 22.27 | Hurt |
| train LM on data until 2009 rather than 2008 | Transformer-XL · NewsQA 2009 questions | F1: 0.47 -> 0.47 | No clear effect |
Closest papers by meaning
- Arrows of Time for Large Language Models (ICML 2024 · similarity 0.90)
- Language Models Are Implicitly Continuous (ICLR 2025 · similarity 0.89)
- Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale (NeurIPS 2025 · similarity 0.88)
- PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs (ICLR 2025 · similarity 0.88)
- Less is More: Local Intrinsic Dimensions of Contextual Language Models (NeurIPS 2025 · similarity 0.87)
- Flexible Language Modeling in Continuous Space with Transformer-based Autoregressive Flows (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.