Research map / Large language models / Language, safety and interpretability
Introducing FOReCAst: The Future Outcome Reasoning and Confidence Assessment Benchmark
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces FOReCAst, a Metaculus-based benchmark for LLM forecasting covering Boolean, timeframe and quantity questions, scoring both prediction accuracy and confidence calibration with crowd-derived gold confidence.
Problem
Existing forecasting benchmards lack confidence assessment, cover few question types, and often use artificial questions rather than real-world ones.
Approach
Gold confidence is derived from log scores of aggregated Metaculus community forecasts, and calibration is measured with a modified Brier score for Boolean questions and CRPS over Gaussian distributions built from confidence for timeframe and quantity questions.
Key results
- Qwen-2.5-7B-Instruct and OLMo-7B-Instruct reach ~0.68 and 0.65 accuracy on Boolean questions, but Llama-3.1-8B fails with 0.08 accuracy and 0.99 Brier.
- Timeframe prediction ADE and CRPS stay near 1.0 for most models; only OLMo-2-7B-Instruct reaches CRPS 0.82.
- Pythia-160m and GPT2-XL achieve very low APE (0.02 and 0.04) yet poor CRPS (0.65 and 0.72), while OLMo-2-7B-Instruct attains the lowest quantity CRPS (0.57).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add instruction tuning (ablation) | OLMo-7B · FOReCAst Boolean Questions | accuracy: 0.21 -> 0.65 | Helped |
| Add instruction tuning (ablation) | OLMo-7B · FOReCAst Boolean Questions | Brier: 0.82 -> 0.38 | Helped |
| Add instruction tuning (ablation) | OLMo-2-7B · FOReCAst Boolean Questions | accuracy: 0.51 -> 0.59 | Helped |
| Add instruction tuning (ablation) | OLMo-2-7B · FOReCAst Boolean Questions | Brier: 0.44 -> 0.41 | Helped |
| Add instruction tuning (ablation) | OLMo-2-7B · FOReCAst Timeframe Prediction | CRPS: 1.00 -> 0.82 | Helped |
| Add instruction tuning (ablation) | OLMo-2-7B · FOReCAst Quantity Estimation | APE: 0.02 -> 0.09 | Hurt |
| Add instruction tuning (ablation) | OLMo-2-7B · FOReCAst Quantity Estimation | CRPS: 0.62 -> 0.57 | Helped |
| Use Bayesian aggregation instead of highest-confidence selection (ablation) | Llama-7B · FOReCAst Boolean Questions | accuracy: 0.57 -> 0.58 | Helped |
| Use Bayesian aggregation instead of highest-confidence selection (ablation) | Llama-7B · FOReCAst Boolean Questions | Brier: 0.52 -> 0.55 | Hurt |
| Use weighted-average confidence aggregation instead of highest-confidence selection (ablation) | Llama-7B · FOReCAst Boolean Questions | Brier: 0.52 -> 0.29 | Helped |
| Use majority-vote aggregation instead of highest-confidence selection (ablation) | Llama-7B · FOReCAst Boolean Questions | Brier: 0.52 -> 0.53 | Hurt |
| Use logit-mean probability aggregation instead of highest-confidence selection (ablation) | Llama-7B · FOReCAst Boolean Questions | Brier: 0.52 -> 0.62 | Hurt |
Surprising findings
- Model size and training-data recency do not reliably predict forecasting or calibration performance.
- Accurate point predictions do not imply well-calibrated confidence in quantity estimation.
Limitations
- Built entirely from Metaculus, limiting diversity of forecasting platforms and formats.
- Gold confidence derived from community forecasts may hide disagreement and carry group biases.
- English-only questions.
Experimental setup
Datasets: FOReCAst (Metaculus-derived questions)
Benchmarks: FOReCAst
Metrics: Accuracy F1 Brier score ADE CRPS APE MAE
Baselines: GPT2 Pythia BLOOM LLaMA-7B OLMo OLMo-2 Qwen 2.5 Llama-3.1-8B
Builds on: ForecastQA AutoCast ExpTime ChaosNLI
Closest papers by meaning
- ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities (ICLR 2025 · similarity 0.88)
- Context is Key: A Benchmark for Forecasting with Essential Textual Information (ICML 2025 · similarity 0.85)
- fev-bench: A Realistic Benchmark for Time Series Forecasting (NeurIPS 2026 · similarity 0.85)
- Consistency Checks for Language Model Forecasters (ICLR 2025 · similarity 0.85)
- ProbTS: Benchmarking Point and Distributional Forecasting across Diverse Prediction Horizons (NeurIPS 2024 · similarity 0.84)
- Conditional Forecasts and Proper Scoring Rules for Reliable and Accurate Performative Predictions (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.