Research map / Large language models / Language, safety and interpretability
Future Language Modeling from Temporal Document History
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces future language modeling, generating texts for a future year from a temporal history of documents, and proposes GPT-2 based models (word frequency bias, temporal contextual bias, doubly contextualized gated bias) evaluated on ACL abstracts.
Problem
Automatically predicting future textual content from a temporal history of documents has received little formal study in ML/NLP; standard language models ignore temporal drift in language.
Approach
Add a learned temporal bias, computed by an LSTM over previous years' word frequencies or contextualized word embeddings, to the softmax of a GPT-2 decoder; the doubly contextualized variant uses sigmoid gating conditioned on the decoder state so the bias is applied only where useful.
Key results
- Doubly contextualized model (Context^2) reaches test PPL 19.81 vs 22.76 for Baseline-all, and test content meteor 20.12 vs 16.22.
- Word frequency model reaches test PPL 20.20 and CM 19.98, improving over Baseline-all on all automatic metrics.
- In human evaluation over 100 abstracts per approach, Context^2 scores 63% average vs 46% for Baseline-all, with higher novelty of topics, problems and methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use gated, doubly contextualized temporal word bias instead of the non-temporal baseline | GPT-2 · ACL abstracts 2021 test | PPL: 22.76 -> 19.81 | Helped |
| Use gated, doubly contextualized temporal word bias instead of the non-temporal baseline | GPT-2 · ACL abstracts 2021 test | CM: 16.22 -> 20.12 | Helped |
| Use gated, doubly contextualized temporal word bias instead of the non-temporal baseline | GPT-2 · ACL abstracts 2021 test | CPL: 102.03 -> 82.43 | Helped |
| Add an LSTM-predicted word-frequency temporal bias instead of the non-temporal baseline | GPT-2 · ACL abstracts 2021 test | CM: 16.22 -> 19.98 | Helped |
| Add an LSTM-predicted word-frequency temporal bias instead of the non-temporal baseline | GPT-2 · ACL abstracts 2021 test | PPL: 22.76 -> 20.20 | Helped |
| Replace the LSTM-predicted bias with previous-year word frequency (ablation) | Frequency · ACL abstracts 2021 test | PPL: 20.20 -> 21.23 | Hurt |
| Replace the LSTM-predicted bias with previous-year word frequency (ablation) | Frequency · ACL abstracts 2021 test | CM: 19.98 -> 19.09 | Hurt |
| Remove the generation-time gate from the temporal contextual model (ablation) | Context² · ACL abstracts 2021 test | PPL: 19.81 -> 23.21 | Hurt |
| Remove the generation-time gate from the temporal contextual model (ablation) | Context² · ACL abstracts 2021 test | CM: 20.12 -> 18.20 | Hurt |
| Additively combine the word frequency and temporal contextual models (ablation) | Future language model · ACL abstracts | overall performance | No clear effect |
| Use gated, doubly contextualized temporal word bias instead of the non-temporal baseline | GPT-2 · Human evaluation of generated ACL abstracts | Avg: 46% -> 63% | Helped |
Surprising findings
- Training only on the most recent years (Baseline-2, Baseline-3) did not beat training on all years in perplexity on test, and the temporal contextual model alone (Context) was worse than the baseline on test perplexity.
Limitations
- Topic-agnostic generation only; evaluated on a single domain (NLP abstracts).
- Generated content may contain factual errors, reuse old ideas, and lack citations.
- Requires fine-tuning the model architecture, so not directly applicable to large LLMs such as GPT-4.
Experimental setup
Datasets: ACL Anthology abstracts 2003-2021
Benchmarks: ACL Anthology abstracts 2020 dev / 2021 test
Metrics: perplexity content perplexity content meteor human evaluation of topic/problem/method clarity and novelty
Baselines: Baseline-all (GPT-2 fine-tuned on all previous years) Baseline-2 Baseline-3 Frequency-NoLSTM
Builds on: GPT-2 RoBERTa LSTM
Closest papers by meaning
- Approaching Human-Level Forecasting with Language Models (NeurIPS 2024 · similarity 0.87)
- LLM Processes: Numerical Predictive Distributions Conditioned on Natural Language (NeurIPS 2024 · similarity 0.86)
- AutoTimes: Autoregressive Time Series Forecasters via Large Language Models (NeurIPS 2024 · similarity 0.86)
- ChatTime: A Unified Multimodal Time Series Foundation Model Bridging Numerical and Textual Data (AAAI 2025 · similarity 0.86)
- Arrows of Time for Large Language Models (ICML 2024 · similarity 0.85)
- Context is Key: A Benchmark for Forecasting with Essential Textual Information (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.