Research map / Large language models / Retrieval and knowledge
ChroKnowledge: Unveiling Chronological Knowledge of Language Models in Multiple Domains
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ChroKnowBench, a multi-domain benchmark of time-variant and time-invariant knowledge, the SliCK-based ChroKnowledge evaluation framework, and ChroKnowPrompt, a chronological prompting method that traverses adjacent years to recover partially known facts.
Problem
LLMs' chronological (time-aware) knowledge is hard to assess because existing temporal evaluations rely on single timestamps and do not capture how knowledge evolves across time spans in multiple domains.
Approach
Extends sampling-based knowledge categorization (SliCK) with a time component and a benchmark that tracks object changes per subject-relation over years, then uses ChroKnowPrompt to iteratively feed object answers from previous and next years as context so the model can verify its answer for the target year.
Key results
- ChroKnowPrompt raises the Known category by an average of 11.5% in the biomedical domain, 2.8% in general, and 3.1% in legal, across models.
- For changed objects in dynamic datasets, ChroKnowPrompt reaches only 10-30% of the performance seen on unchanged objects.
- In the biomedical domain, the total span (previous plus next) nearly doubles the score of previous span alone (12.0 vs. 6.7).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply ChroKnowPrompt to unchanged objects | LLMs evaluated in the paper · Biomedical | Known category: +11.5% | Helped |
| Apply ChroKnowPrompt to unchanged objects | LLMs evaluated in the paper · General | Known category: +2.8% | Helped |
| Apply ChroKnowPrompt to unchanged objects | LLMs evaluated in the paper · Legal | Known category: +3.1% | Helped |
| Add the next chronological span to the previous span (ablation) | ChroKnowPrompt · Biomedical | Score: 6.7 -> 12.0 | Helped |
| Add the next chronological span to the previous span (ablation) | ChroKnowPrompt · General | Score: 1.8 -> 2.8 | Helped |
| Apply ChroKnowPrompt | mpt-7b-chat · Chat models | Average increase: 11.4 | Helped |
| Apply ChroKnowPrompt | Pythia-Chat-Base-7B · Chat models | Knowledge recall improvement | No clear effect |
| Apply ChroKnowPrompt to changed objects in dynamic datasets | LLMs evaluated in the paper · Dynamic datasets across domains | Performance relative to unchanged-object cases: 10–30% | Hurt |
| Use chronological prompting on legal data | LLMs evaluated in the paper · Legal dynamic dataset | Knowledge recall | Hurt |
Surprising findings
- Chronological prompting is not particularly effective for chat models; only mpt-7b-chat gains comparably to Llama-2-7b-chat-hf, while Pythia-Chat-Base-7B shows almost no improvement.
- Generation performance declines on recent years while MCQA and TF rise over time, so template choice changes temporal trends.
Limitations
- ChroKnowPrompt performs poorly on changed objects in dynamic datasets and on unstructured legal text.
- Object-level focus may overlook broader event interdependencies compared with temporal knowledge graphs.
- Evaluation relies on fuzzy string matching against labels, which can misjudge complex objects.
Experimental setup
Datasets: Wikidata UMLS Code of Federal Regulations (CFR) CSKG Math-KG
Benchmarks: ChroKnowBench
Metrics: Correct / Partial Correct / Incorrect rate Known / Partial Known / Cut-off / Unknown categories Chrono-correct rate
Baselines: Llama-2-7b-chat-hf mpt-7b-chat Pythia-Chat-Base-7B nemotron-3-8b-chat-4k-sft-hf
Builds on: SliCK BioLAMA Wikidata CSKG Math-KG
Closest papers by meaning
- History Matters: Temporal Knowledge Editing in Large Language Model (AAAI 2024 · similarity 0.89)
- TimE: A Multi-level Benchmark for Temporal Reasoning of LLMs in Real-World Scenarios (NeurIPS 2025 · similarity 0.87)
- Large Language Models' Expert-level Global History Knowledge Benchmark (HiST-LLM) (NeurIPS 2024 · similarity 0.87)
- KoLA: Carefully Benchmarking World Knowledge of Large Language Models (ICLR 2024 · similarity 0.87)
- Episodic Memories Generation and Evaluation Benchmark for Large Language Models (ICLR 2025 · similarity 0.87)
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.