Research map / Large language models / Language, safety and interpretability
Martingale Score: An Unsupervised Metric for Bayesian Rationality in LLM Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the Martingale Score, an unsupervised regression-based metric (OLS slope of belief update on prior belief) to quantify belief entrenchment in LLM reasoning, applied across forecasting, r/ChangeMyView, and OpenReview with CoT and debate.
Problem
LLM reasoning may entrench prior beliefs (confirmation bias) rather than update rationally on new evidence, and existing measures need ground truth or synthetic setups, so open-ended reasoning failures are hard to detect.
Approach
Operationalizes belief entrenchment as a violation of the Martingale property: the OLS slope of belief update (posterior minus prior) on prior belief, which should be zero under rational Bayesian updating; the score is unsupervised and domain-agnostic, and is validated against Brier score where ground truth exists.
Key results
- Martingale Scores are positive in 51 of 54 CoT setups, indicating widespread belief entrenchment; mean M for CMV-CoT is 0.103 vs 0.037 for Forecasting-CoT.
- Martingale Score positively correlates with Brier score in Forecasting, with a score of 0.04 associated with performance worse than random guessing (Brier > 0.25).
- DeepSeek R1 shows significantly lower belief entrenchment than all other tested models, and debate reduces entrenchment relative to CoT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use debate instead of CoT | LLMs across evaluated models and domains · Forecasting, ChangeMyView, and OpenReview | Martingale Score | Helped |
| Use a prior-conforming prompt instead of no system prompt (ablation) | LLMs · Forecasting | Martingale Score: 0.075 \pm0.014 -> 0.082 \pm0.018 | Hurt |
| Use a critical-thinking prompt instead of no system prompt (ablation) | LLMs · Forecasting | Martingale Score: 0.075 \pm0.014 -> 0.072 \pm0.018 | Helped |
| Use a critical-thinking prompt instead of no system prompt (ablation) | LLMs · ChangeMyView and OpenReview | Martingale Score | No clear effect |
| Add a prior-conforming prompt (ablation) | GPT-4o (May 7), CoT · Forecasting | Martingale Score: +0.0018 -> +0.0896^{\ast} | Hurt |
| Add a prior-conforming prompt (ablation) | DeepSeek R1, CoT · Forecasting | Martingale Score: +0.0207^{\ast} -> +0.0450^{\ast} | Hurt |
| Evaluate in ChangeMyView rather than Forecasting | same model, prompt, and CoT setup · ChangeMyView vs Forecasting | Martingale Score: 0.037 \pm0.011 -> 0.103 \pm0.013 | Hurt |
| Evaluate in OpenReview rather than Forecasting | LLMs · OpenReview vs Forecasting | Martingale Score | Hurt |
| Increase the absolute value of the Martingale Score | reasoning setups · Forecasting | Brier Score: Martingale Score 0.0: Brier Score <0.25; Martingale Score 0.04: worse than 0.250 | Hurt |
| Use DeepSeek R1 rather than the other tested models | LLMs · Across evaluated domains | Martingale Score | Helped |
| Use a prior-conforming prompt rather than no prompt (ablation) | LLMs · Across evaluated setups | Martingale Score: \overline{M}_{\text{No-prompt}}=0.075 \pm0.014 -> \overline{M}_{\text{Prior-conforming}}=0.082 \pm0.018 | Hurt |
| Switch from GPT-4o (Apr 30) no-prompt CoT to prior-conforming CoT (ablation) | GPT-4o (Apr 30) · Forecasting | Martingale Score: +0.016 -> +0.080 | Hurt |
Surprising findings
- Belief entrenchment persists even under no-system-prompt and critical-thinking prompts, not only under prior-conforming prompts.
- No significant Martingale-Brier correlation is found for OpenReview, attributed to noisy community-voted ground truth.
Limitations
- No significant correlation between Martingale Score and accuracy in OpenReview because community-voted ground truth is noisy.
- Only internal reasoning is studied; external evidence search and reinforced reasoning models are not systematically evaluated.
- Belief is measured via LLM-judge expressed beliefs rather than internal beliefs, and judges show only moderate agreement with humans.
Experimental setup
Datasets: Metaculus forecasting questions Polymarket forecasting questions r/ChangeMyView posts ICLR OpenReview submissions
Benchmarks: Metaculus Polymarket r/ChangeMyView OpenReview ICLR
Metrics: Martingale Score Brier score Pearson correlation Spearman correlation
Baselines: GPT-4o DeepSeek R1 DeepSeek V3 Gemini 2.0 Flash Llama 4 Scout Llama 4 Maverick
Builds on: chain-of-thought prompting debate
Closest papers by meaning
- From Drift to Coherence: Stabilizing Beliefs in LLMs (ICML 2026 · similarity 0.90)
- Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs (ICLR 2026 · similarity 0.89)
- Operads for compositional reasoning in LLMs (NeurIPS 2026 · similarity 0.88)
- Evaluating LLMs When They Do Not Know the Answer: Statistical Evaluation of Mathematical Reasoning via Comparative Signals (ICML 2026 · similarity 0.88)
- RATT: A Thought Structure for Coherent and Correct LLM Reasoning (AAAI 2025 · similarity 0.88)
- A Theoretical Study on Bridging Internal Probability and Self-Consistency for LLM Reasoning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.