Research map / Large language models / Language, safety and interpretability
LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arena
Qingchuan Yang, Simon Mahns, Sida Li, Anri Gu, Jibang Wu, Haifeng Xu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add both market data and news sources to the prompt (ablation) | LLM forecasters · Prophet Arena subset (100 events) | Brier score | Improved |
| add news sources on top of market data (ablation) | LLM forecasters · Prophet Arena subset (100 events) | Brier score variance | Improved |
| market data only vs market data plus sources (ablation) | LLM forecasters · Prophet Arena subset (100 events) | Brier score | No clear effect |
| add sources for entertainment and sports events (ablation) | LLM forecasters · Prophet Arena subset (100 events) | Brier score | No clear effect |
| reasoning model GPT-5 instead of Gemini 2.5 Flash | reasoning quality · Prophet Arena reasoning evaluation | reasoning synthesis gap: 0.95 | Improved |
| reasoning model GPT-5 instead of Gemini 2.5 Flash | reasoning quality · Prophet Arena reasoning evaluation | evidence extraction gap: 0.00 | No clear effect |
| forecast further from resolution (use market baseline vs LLM) | Market Baseline vs LLMs · Prophet Arena | Brier score | Mixed |
Closest papers by meaning
- Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle (ICML 2025 · similarity 0.89)
- Approaching Human-Level Forecasting with Language Models (NeurIPS 2024 · similarity 0.87)
- A Theory of Training Profit-Optimal LLMs (NeurIPS 2026 · similarity 0.87)
- LLM Processes: Numerical Predictive Distributions Conditioned on Natural Language (NeurIPS 2024 · similarity 0.87)
- Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.87)
- Consistency Checks for Language Model Forecasters (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
