Research map / Large language models / Retrieval and knowledge
Approaching Human-Level Forecasting with Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a retrieval-augmented LM forecasting system (query generation, news retrieval, relevance filtering, summarization, scratchpad reasoning, trimmed-mean ensembling) plus a GPT-4 fine-tuned on self-selected reasonings, and evaluates on a new dataset of 5 competitive platforms.
Problem
Whether language models can forecast binary future events at the level of competitive human forecasters, measured by Brier score on questions published after LM knowledge cut-offs.
Approach
Combines LM-driven news retrieval (query decomposition, LM relevance rating and summarization) with ensembled scratchpad reasoning from GPT-4 and a self-supervised fine-tuned GPT-4 trained only on reasonings that beat the crowd, then aggregates forecasts with a trimmed mean.
Key results
- The full system achieves a Brier score of 0.179 on the test set vs. 0.149 for the human crowd, with accuracy 71.5% vs. 77.0%.
- Blending the system with the crowd (4:1 crowd weight) improves Brier score from 0.149 to 0.146 on the full test set.
- Under selective criteria (all three conditions met), the system reaches Brier 0.240 vs. crowd 0.247, outperforming the crowd; the best zero-shot baseline is 0.208 (GPT-4-1106-Preview).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add retrieval, optimized reasoning, and fine-tuning to the GPT-4-1106-Preview baseline | GPT-4-1106-Preview · All Platforms test set | Brier score: 0.208 -> $.179_{.003}$ | Helped |
| Use the full system instead of the human crowd | Forecasting system · All Platforms test set | Brier score: $.149_{.003}$ -> $.179_{.003}$ | Hurt |
| Combine system predictions with crowd predictions using 4x crowd weight | Human crowd predictions · All Platforms test set | Brier score: $.149$ -> $.146$ | Helped |
| Forecast only when the crowd prediction is between .3 and .7 | Full forecasting system · All Platforms test set, crowd prediction between .3 and .7 | Brier score: $.240$ -> $.238$ | Helped |
| Forecast only when crowd uncertainty, early retrieval, and at least 5 relevant articles all apply | Full forecasting system · All Platforms test set, all three selective criteria | Brier score: $.247$ -> $.240$ | Helped |
| Replace fine-tuned GPT-4 with fine-tuned GPT-3.5 (ablation) | Full forecasting system · All Platforms test set | Brier score: $.179$ -> $.182$ | Hurt |
| Remove the base GPT-4 ensemble and use fine-tuned GPT-4-0613 (ablation) | Full forecasting system · All Platforms test set | Brier score: $.179_{.003}$ -> $.182_{.002}$ | Hurt |
| Use fine-tuned GPT-3.5 together with base GPT-4 instead of fine-tuned GPT-4 (ablation) | Full forecasting system · All Platforms test set | Brier score: $.179_{.003}$ -> $.181_{.002}$ | Hurt |
| Use fine-tuned GPT-3.5 instead of the full system's fine-tuned GPT-4 and base GPT-4 ensemble (ablation) | Full forecasting system · All Platforms test set | Brier score: $.179_{.003}$ -> $.183_{.002}$ | Hurt |
| Remove fine-tuning (ablation) | Full forecasting system · All Platforms test set | Brier score: $.179$ -> $.186$ | Hurt |
| Remove fine-tuning and news retrieval (ablation) | Full forecasting system · All Platforms test set | Brier score: $.179_{.003}$ -> $.206_{.002}$ | Hurt |
| Add news retrieval while keeping the base GPT-4 model and no fine-tuning (ablation) | Base GPT-4-1106-Preview system · All Platforms test set | Brier score: $.206_{.002}$ -> $.186_{.002}$ | Helped |
Surprising findings
- Without retrieval or fine-tuning, the system falls back to baseline performance (Brier 0.206), and fine-tuning a less capable GPT-3.5 yields nearly the same performance as fine-tuned GPT-4.
- Base LMs are not naturally good at forecasting; most score near or worse than the 0.25 random baseline in zero-shot and scratchpad settings.
Limitations
- Systems tend to hedge predictions, underperforming the crowd on questions where the crowd is highly confident.
- Test set is small per platform (e.g., INFER has 4 test questions); standard errors assume i.i.d. data though data are time-series.
- Relies on proprietary APIs (GPT-4, news APIs), limiting reproducibility and cost.
Experimental setup
Datasets: Metaculus Good Judgment Open INFER Polymarket Manifold
Benchmarks: forecasting test set of 914 binary questions from 5 platforms (post June 1, 2023)
Metrics: Brier score accuracy RMS calibration error standard error
Baselines: GPT-4-1106-Preview zero-shot GPT-4-1106-Preview scratchpad Claude-2.1 Gemini-Pro Llama-2-70B Mistral-8x7B-Instruct human crowd aggregate Base GPT-4 no IR
Builds on: Autocast chain-of-thought prompting universal self-consistency STaR-style self-training
Closest papers by meaning
- Context is Key: A Benchmark for Forecasting with Essential Textual Information (ICML 2025 · similarity 0.89)
- From News to Forecast: Integrating Event Analysis in LLM-Based Time Series Forecasting with Reflection (NeurIPS 2024 · similarity 0.89)
- TS-RAG: Retrieval-Augmented Generation based Time Series Foundation Models are Stronger Zero-Shot Forecaster (NeurIPS 2025 · similarity 0.87)
- A decoder-only foundation model for time-series forecasting (ICML 2024 · similarity 0.87)
- ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities (ICLR 2025 · similarity 0.87)
- Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.