Research map / Large language models / Retrieval and knowledge

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle

ICML 2025 · Large language models › Retrieval and knowledge · approach: Long-context and personalized retrieval · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Daily Oracle, a continuously updated benchmark that auto-generates True/False and multiple-choice future-event forecasting questions from daily news, and uses it to show LLM accuracy degrades over 2020-2024 in closed-book, RAG, and gold-article settings.

Problem

Static LLM benchmarks become outdated and leak into training data, and they lack a temporal dimension for measuring how LLM performance changes over time; forecasting ability over time is understudied.

Approach

Automatically generate forecasting QA pairs from daily news (resolution date = article publication date) with an LLM pipeline (summary, QA generation, distractor generation, rubric-based filtering), so the benchmark stays current and supports a temporal analysis of model accuracy.

Key results

• Average accuracy falls from 64.68% to 50.74% on TF questions (-21.55%) and from 58.30% to 51.69% on MC questions (-11.33%) from 2020 to 2024 in the closed-book setting.
• Gold-article setting reaches around 90% accuracy for most models, yet most models still show declining trends over time.
• RAG with recent news improves accuracy but the downward degradation trend persists across all retrieval cutoffs.

Measured effects reported in the paper

Surprising findings

• Even with access to gold articles containing the answer, most models still show declining accuracy over time, suggesting outdated internal representations.
• Claude-3.5-Sonnet's constrained open-book performance lags its closed-book performance, as retrieved information can confuse a model with robust world knowledge.

Limitations

• Generated questions and distractors may carry biases from the outdated LLM used to generate them, reducing long-run reliability unless the generator is upgraded.
• Question generation from news focuses on events that occurred, potentially overlooking events that never happen.
• Time horizon after model cutoffs is too short for thorough analysis of post-cutoff trends; human agreement (Fleiss' Kappa 0.26) is only fair.

Experimental setup

Datasets: Common Crawl News Newspaper3k scraped news Daily Oracle generated QA pairs

Benchmarks: Daily Oracle

Metrics: accuracy year-over-year accuracy change refusal rate

Baselines: GPT-4 GPT-3.5 Claude-3.5-Sonnet Mixtral-8x7B Mistral-7B Llama-3-8B Qwen-2-7B Gemma-2-2B

Builds on: RAG AutoCast StreamingQA

Closest papers by meaning

• tinyBenchmarks: evaluating LLMs with fewer examples (ICML 2024 · similarity 0.89)
• Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning (ICLR 2025 · similarity 0.87)
• Evaluating language models as risk scores (NeurIPS 2024 · similarity 0.87)
• Approaching Human-Level Forecasting with Language Models (NeurIPS 2024 · similarity 0.87)
• LLMEval: A Preliminary Study on How to Evaluate Large Language Models (AAAI 2024 · similarity 0.86)
• Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG (ICLR 2025 · similarity 0.86)