Research map / Large language models / Agents, code and math
TimE: A Multi-level Benchmark for Temporal Reasoning of LLMs in Real-World Scenarios
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces TimE, a multi-level benchmark (38,522 QA pairs, 11 subtasks, three sub-datasets from Wikidata, news, and dialogues) plus a human-verified TimE-Lite subset for evaluating LLM temporal reasoning in real-world settings.
Problem
Existing temporal reasoning benchmarks focus on simplified scenarios (short texts, basic commonsense) and do not cover intensive temporal information, fast-changing events, or complex temporal dependencies in long social interactions.
Approach
A three-level, 11-subtask benchmark spanning knowledge-intensive (Wikidata-derived temporal knowledge graphs), dynamic news (temporal complex events with RAG), and very-long multi-session dialogues, with a manually verified TimE-Lite subset and multiple-choice questions with misleading options built via STARC.
Key results
- o3-mini reaches only 52.62% on Order Reasoning and 54.34% on Co-temporality in TimE-Wiki, while Level-1 tasks approach 80% on four tasks.
- All models score at or below 30% on the TimE-News Timeline task, which requires ordering three events.
- Deepseek-R1-Distill-Qwen-14B outperforms Qwen2.5-14B-Instruct by 24.44% on Order Compare in TimE-Dial, but underperforms by 8.16% on Localization in TimE-Wiki.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use test-time-scaled Deepseek-R1-Distill-Qwen-14B instead of Qwen2.5-14B-Instruct | Qwen2.5-14B-Instruct · TimE-Dial | Order Compare: +24.44% | Helped |
| Use test-time-scaled Deepseek-R1-Distill-Qwen-14B instead of Qwen2.5-14B-Instruct | Qwen2.5-14B-Instruct · TimE-Dial | Duration Compare: +11.33% | Helped |
| Use test-time-scaled Deepseek-R1-Distill-Qwen-14B instead of Qwen2.5-14B-Instruct | Qwen2.5-14B-Instruct · TimE-Dial | Counterfactual: +12.0% | Helped |
| Use test-time-scaled Deepseek-R1-Distill-Qwen-14B instead of Qwen2.5-14B-Instruct | Qwen2.5-14B-Instruct · TimE-Wiki | Extract: by 3.36% | Hurt |
| Use test-time-scaled Deepseek-R1-Distill-Qwen-14B instead of Qwen2.5-14B-Instruct | Qwen2.5-14B-Instruct · TimE-Wiki | Localization: by 8.16% | Hurt |
| Use Hybrid instead of BM25 or Vector retrieval | GPT-4o with TimE-News RAG · TimE-News | Timeline: over 10% lower | Hurt |
Surprising findings
- Test-time scaling does not consistently help time retrieval and event localization; reasoning models can overthink after retrieval errors.
- Small vanilla models score below 10% on the Timeline task in TimE-Wiki and TimE-Dial.
Limitations
- Many subtasks rely on LLM-synthesized (DeepSeek-V3/R1) questions and contexts, with only a manually verified subset of about 940 items.
- Free-form answers have synonym issues, so the benchmark relies on multiple-choice formats for News and Dial.
Experimental setup
Datasets: Wikidata TempLAMA-style SLING-parsed temporal facts Temporal complex events news dataset LoCoMo RealTalk
Benchmarks: TimE-Wiki TimE-News TimE-Dial TimE-Lite
Metrics: Exact Match F1 macro F1 (option-level) Word-level Similarity (annotation consistency)
Baselines: Llama-3.1-8B-Instruct Qwen2.5-72B-Instruct QwQ-32B Deepseek-R1 GPT-4o OpenAI o3-mini Deepseek-V3
Builds on: TempLAMA SLING STARC LoCoMo RealTalk
Closest papers by meaning
- Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning (ICLR 2025 · similarity 0.94)
- NeSTR: A Neuro-Symbolic Abductive Framework for Temporal Reasoning in Large Language Models (AAAI 2026 · similarity 0.88)
- ReXTime: A Benchmark Suite for Reasoning-Across-Time in Videos (NeurIPS 2024 · similarity 0.88)
- A Comprehensive Evaluation on Event Reasoning of Large Language Models (AAAI 2025 · similarity 0.88)
- Bag of Tricks for Inference-time Computation of LLM Reasoning (NeurIPS 2025 · similarity 0.88)
- Multilingual LLMs Inherently Reward In-Language Time-Sensitive Semantic Alignment for Low-Resource Languages (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.