Research map / Large language models / AI and society
ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ForecastBench, a continuously updated benchmark of ~1,000 unresolved forecasting questions, and compares 17+ LLMs against public and superforecaster medians, finding the best LLMs trail superforecasters.
Problem
Existing LLM forecasting evaluations are static, built on resolved questions that fall within model knowledge cut-offs, which risks contamination and obsolescence; there is no standardized, live benchmark for comparing ML forecasters with human forecasters.
Approach
A dynamic benchmark that nightly draws questions from prediction markets and real-world datasets, only includes questions unresolved at submission time, and scores LLMs against human public and superforecaster medians on a shared question set with freeze values and combination questions.
Key results
- Superforecaster median achieves overall Brier 0.096 vs. 0.121 for the general public and 0.122 for the top LLM (Claude-3.5-Sonnet with freeze values and scratchpad) on the 200-question human set.
- On the full LLM question set including combination questions, superforecasters (Brier 0.076) significantly outperform the public (0.096) and the top LLM (GPT-4o, 0.130).
- Extrapolating the Chatbot Arena trend suggests LLMs would match superforecaster performance at an Arena score of about 1406.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the superforecaster median instead of the public median | ForecastBench human subset · ForecastBench human subset (200 questions; 7-, 30-, 90-, 180-day horizons) | Brier score: 0.121 -> 0.096 | Helped |
| Provide news retrieval in addition to freeze values | Claude-3.5 Sonnet with scratchpad prompt · ForecastBench human subset (200 questions; 7-, 30-, 90-, 180-day horizons) | Brier score: 0.122 -> 0.127 | Hurt |
| Provide news retrieval in addition to freeze values | GPT-4o with scratchpad prompt · ForecastBench full LLM question set (1,000 questions; 7-, 30-, 90-, 180-day horizons) | Brier score: 0.128 -> 0.137 | Hurt |
| Use a scratchpad prompt instead of zero-shot, with freeze values | Claude-3.5 Sonnet · ForecastBench human subset (200 questions; 7-, 30-, 90-, 180-day horizons) | Brier score: 0.131 -> 0.122 | Helped |
| Add combination questions to the evaluation | LLM forecasting systems · ForecastBench combination questions | Brier score: 0.076; 0.096; 0.130 | Mixed |
| Use superforecasters rather than the top LLM | ForecastBench combination questions · ForecastBench combination questions | Brier score: 0.130 -> 0.076 | Helped |
| Use a higher Chatbot Arena score | LLM forecasting models · ForecastBench overall | Brier score | Helped |
Surprising findings
- The top-performing models without access to crowd forecasts on market questions were noticeably less accurate (Brier 0.136), while access to news retrieval did not improve performance.
- LLMs perform markedly worse on combination questions, widening the gap to human forecasters.
Limitations
- Human comparison uses only 200 standard questions and a median aggregate, limiting statistical power for some comparisons.
- Market questions are scored against crowd forecasts before resolution, which is an imperfect proxy for ground truth.
- Human combination-question performance is bounded by assuming independence between components, underestimating human accuracy.
Experimental setup
Datasets: Manifold Markets Metaculus Polymarket ACLED DBnomics FRED Wikipedia Yahoo! Finance RAND Forecasting Initiative
Benchmarks: ForecastBench
Metrics: Brier score bootstrapped p-value confidence interval
Baselines: Claude-3.5-Sonnet GPT-4o GPT-4-Turbo GPT-4 Gemini-1.5-Pro Claude-3-Opus Qwen1.5-110B-Chat Public median forecast Superforecaster median forecast
Builds on: Brier score chain-of-thought prompting retrieval-augmented generation
Closest papers by meaning
- Consistency Checks for Language Model Forecasters (ICLR 2025 · similarity 0.88)
- Introducing FOReCAst: The Future Outcome Reasoning and Confidence Assessment Benchmark (NeurIPS 2025 · similarity 0.88)
- ProbTS: Benchmarking Point and Distributional Forecasting across Diverse Prediction Horizons (NeurIPS 2024 · similarity 0.87)
- Approaching Human-Level Forecasting with Language Models (NeurIPS 2024 · similarity 0.87)
- Context is Key: A Benchmark for Forecasting with Essential Textual Information (ICML 2025 · similarity 0.85)
- fev-bench: A Realistic Benchmark for Time Series Forecasting (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.