Research map / Large language models / Language, safety and interpretability
Consistency Checks for Language Model Forecasters
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes arbitrage- and frequentist-based consistency metrics for LLM forecasters, builds a pipeline of logically related binary questions, shows consistency (especially Cond) correlates with Brier score, and releases a 2028-resolving consistency benchmark plus an ArbitrageForecaster wrapper.
Problem
Evaluating LLM forecasters is hard when ground truth resolves only in the future; the paper asks whether logical consistency across related forecasting questions can serve as an instantaneous proxy for forecasting accuracy.
Approach
Consistency violations are measured via an arbitrage metric (minimum guaranteed profit for an arbitrageur betting against a log-market-maker) and a frequentist hypothesis-test metric, applied uniformly to 10 logical checks; the arbitrage wrapper ArbitrageForecaster is used to test whether optimizing consistency improves forecasts.
Key results
- Cond arbitrage metric correlates with Brier score at R=0.92 on the scraped dataset (0.71 on NewsAPI).
- Aggregated frequentist metric correlates with Brier score at 0.85 on scraped questions.
- ArbitrageForecaster improves consistency on the checks it is arbitraged on but does not reliably improve Brier score or held-out checks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Aggregate consistency violations across checks as a forecasting-performance proxy | LLM forecasters · Scraped forecasting questions | Correlation with Brier score: 0.62; 0.85 | Helped |
| Aggregate consistency violations across checks as a forecasting-performance proxy | LLM forecasters · NewsAPI questions | Correlation with Brier score: 0.49; 0.66 | Helped |
| Measure conditional-probability consistency with the Cond check | LLM forecasters · Scraped forecasting questions | Correlation with Brier score: 0.92; 0.87 | Helped |
| Measure conditional-probability consistency with the Cond check | LLM forecasters · NewsAPI questions | Correlation with Brier score: 0.71; 0.69 | Helped |
| Measure consistency with the AndOr check | LLM forecasters · NewsAPI questions | Correlation with Brier score: 0.02; 0.06 | No clear effect |
| Arbitrage a forecaster on a consistency check | ArbitrageForecaster using gpt-4o-mini · The consistency check being optimized | Consistency violation | Helped |
| Increase recursive arbitrage depth on the optimized check | ArbitrageForecaster using gpt-4o-mini · The consistency check being optimized | Consistency violation | Helped |
| Arbitrage on Negation and Paraphrase together | ArbitrageForecaster using gpt-4o-mini · Negation and Paraphrase | Consistency violation | Helped |
| Arbitrage a forecaster on one check | ArbitrageForecaster using gpt-4o-mini · Other held-out consistency checks | Consistency violation | No clear effect |
| Arbitrage a forecaster on one check | ArbitrageForecaster using gpt-4o-mini · Ground-truth forecasting performance | Brier score | No clear effect |
Surprising findings
- Even o1-mini violates the consistency threshold on a large fraction of tuples (e.g., 58% for Negation on scraped questions).
- Negation and And-type correlations with Brier score flip sign on the NewsAPI dataset, so consistency-accuracy correlation is not uniform across checks.
Limitations
- Tested only a limited set of forecasters, so trends may not generalize to future LLMs.
- Brier score depends on training data cutoff, confounding the consistency-accuracy correlation.
- ArbitrageForecaster experiments limited to few checks due to API cost.
Experimental setup
Datasets: Metaculus questions Manifold Markets questions NewsAPI articles
Benchmarks: scraped forecasting questions (Metaculus/Manifold) NewsAPI-derived questions 2028 long-horizon consistency benchmark
Metrics: arbitrage violation frequentist violation Brier score Pearson correlation violation fraction
Baselines: gpt-4o-mini claude-3.5-sonnet llama-3.1-405B o1-mini
Builds on: consistency check framework market scoring rules Brier score
Closest papers by meaning
- ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities (ICLR 2025 · similarity 0.88)
- Approaching Human-Level Forecasting with Language Models (NeurIPS 2024 · similarity 0.87)
- Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle (ICML 2025 · similarity 0.86)
- From Drift to Coherence: Stabilizing Beliefs in LLMs (ICML 2026 · similarity 0.85)
- CruxBench: A Benchmark of Information Discovery (NeurIPS 2026 · similarity 0.85)
- Introducing FOReCAst: The Future Outcome Reasoning and Confidence Assessment Benchmark (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.