Research map / Large language models / Agents, code and math
Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Test of Time, a synthetic temporal-reasoning benchmark (semantics via graph-based temporal facts, arithmetic via crowd-sourced time-calculation questions) and evaluates Claude-3-Sonnet, GPT-4, and Gemini 1.5 Pro on it.
Problem
Existing temporal reasoning benchmarks rely on knowledge-graph facts about well-known entities or anonymized real-world data, so LLMs can exploit parametric knowledge or shortcuts, and they under-cover temporal graph structures and time arithmetic.
Approach
Decouple temporal reasoning into semantics/logic (synthetic temporal graphs from random graph generators plus anonymized Wikidata ego-graphs, with eight question types) and arithmetic (crowd-sourced, functionalized time-calculation problems), so models must reason over presented facts rather than memorized knowledge.
Key results
- GPT-4 accuracy ranged from 20.33% on complete graphs to 90.83% on AWE graphs on the synthetic semantics task, with Gemini 1.5 Pro having the best average rank (1.43 by graph, 1.44 by question type).
- Presenting facts sorted by target entity then start time (TargetAndStartTime) gave the highest accuracy for all models, while shuffled order was lowest.
- Timeline questions were the hardest category; on the arithmetic task all models struggled most with Duration questions (GPT-4 16.00%, Gemini 13.50%, Claude 15.00%) while Timezone was easiest (up to 90%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AWE rather than complete graph structure | GPT-4 · ToT synthetic | accuracy: 40.25 -> 92.00 | Helped |
| Increase the number of edges | LLMs on ToT synthetic · ToT synthetic | accuracy | Hurt |
| Increase the number of nodes | LLMs on ToT synthetic · ToT synthetic | accuracy | Hurt |
| Sort facts by target entity and start time instead of shuffling (ablation) | LLMs on ToT synthetic · ToT synthetic | accuracy: 45.71 -> 73.57 | Helped |
| Present facts in shuffled rather than TargetAndStartTime order (ablation) | LLMs on ToT synthetic · ToT synthetic | accuracy: 73.57 -> 45.71 | Hurt |
| Use TargetAndStartTime ordering rather than shuffle (ablation) | Gemini 1.5 Pro on expanded ToT synthetic · ToT synthetic, expanded set | accuracy | Helped |
| Ask a single-fact question rather than a multi-fact question | LLMs on ToT synthetic · ToT synthetic | accuracy | Helped |
| Ask EventAtWhatTime rather than EventAtTimeT | LLMs on ToT synthetic · ToT synthetic | accuracy | Helped |
| Evaluate on Duration rather than Timezone arithmetic questions | LLMs on ToT arithmetic · ToT arithmetic | accuracy | Hurt |
| Use larger synthetic evaluation set | Gemini 1.5 Pro · ToT synthetic | accuracy: 88.72 | Mixed |
Surprising findings
- Fact ordering alone changes LLM accuracy substantially, with shuffled facts performing worst, despite facts being independent.
- LLMs performed better on anonymized Wikidata graphs than on synthetic random graphs, and GPT-4 accuracy more than doubled between complete and AWE graphs, suggesting possible reliance on parametric knowledge.
Limitations
- Focuses only on single-sentence temporal anchoring; multi-sentence or multi-document temporal reasoning not covered.
- Excludes static facts without time anchors, so benchmark covers explicit temporal facts only.
- Evaluated only three proprietary frontier models with no open-weight models.
Experimental setup
Datasets: Wikidata (anonymized ego-graphs for AWE) YAGO11k (related work only)
Benchmarks: Test of Time synthetic semantics dataset (ToT-Semantic) Test of Time arithmetic dataset (ToT-Arithmetic)
Metrics: accuracy precision recall average rank
Baselines: Claude-3-Sonnet GPT-4 Gemini 1.5 Pro GPT-4 Turbo
Builds on: TGQA TempTabQA NetworkX random graph models
Closest papers by meaning
- TimE: A Multi-level Benchmark for Temporal Reasoning of LLMs in Real-World Scenarios (NeurIPS 2025 · similarity 0.94)
- Bag of Tricks for Inference-time Computation of LLM Reasoning (NeurIPS 2025 · similarity 0.90)
- GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (ICLR 2025 · similarity 0.89)
- A State-Transition Framework for Efficient LLM Reasoning (ICLR 2026 · similarity 0.89)
- A Theoretical Study on Bridging Internal Probability and Self-Consistency for LLM Reasoning (NeurIPS 2025 · similarity 0.88)
- Meaningful Learning: Enhancing Abstract Reasoning in Large Language Models via Generic Fact Guidance (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.