Research map / Large language models / Agents, code and math
Reasoning over Uncertain Text by Generative Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces BLInD, a synthetic dataset of Bayesian-network-based probabilistic reasoning over text, and prompting methods (PAL, Monte Carlo, ProbLog, with number extraction and graph generation subtasks) that map problems to symbolic solvers, evaluated on GPT-3.5, GPT-4, and Llama3-70B.
Problem
LLMs struggle to perform probabilistic reasoning over text that explicitly states probabilities, such as Bayesian network queries expressed in natural language.
Approach
Map natural-language probabilistic contexts to formal representations (Python arithmetic, Monte Carlo direct-sampling code, or ProbLog programs), with optional subtasks that first extract CPT numbers and the dependency graph, and execute them with external solvers.
Key results
- ProbLog with GPT-4 reaches 98% average accuracy on BLInD (V2-10), versus 24% for GPT-4 with zero-shot CoT.
- Monte Carlo with Graph Generation on GPT-4 reaches 95% average accuracy, including 85-99% on larger networks.
- On the adapted CLADDER set, GPT-4 with PAL, Monte Carlo, or ProbLog reaches 96-98% accuracy, versus 65% for CoT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Map probabilistic questions to ProbLog instead of direct COT prompting | GPT3.5 · BLInD V₂₋₁₀ | accuracy: 12 -> 62 | Helped |
| Map probabilistic questions to ProbLog instead of direct COT prompting | Llama3 · BLInD V₂₋₁₀ | accuracy: 22 -> 89 | Helped |
| Map probabilistic questions to ProbLog instead of direct COT prompting | GPT4 · BLInD V₂₋₁₀ | accuracy: 29 -> 98 | Helped |
| Add Number Extraction to PAL (ablation) | GPT3.5 · BLInD V₂₋₁₀ | accuracy: 19 -> 29 | Helped |
| Add Number Extraction to PAL (ablation) | Llama3 · BLInD V₂₋₁₀ | accuracy: 40 -> 48 | Helped |
| Add Number Extraction to PAL (ablation) | GPT4 · BLInD V₂₋₁₀ | accuracy: 48 -> 49 | No clear effect |
| Add Graph Generation to Monte Carlo (ablation) | GPT3.5 · BLInD V₂₋₁₀ | accuracy: 46 -> 49 | Helped |
| Add Graph Generation to Monte Carlo (ablation) | GPT4 · BLInD V₂₋₁₀ | accuracy: 94 -> 95 | Helped |
| Add Graph Generation to Monte Carlo (ablation) | Llama3 · BLInD V₂₋₁₀ | accuracy: 87 -> 82 | Hurt |
| Add Graph Generation to Monte Carlo (ablation) | GPT3.5 and GPT4 · CLADDER hard tests adaptation | accuracy: GPT3.5: 75 -> 75; GPT4: 98 -> 97 | No clear effect |
| Use Monte Carlo inference instead of PAL | GPT3.5 · BLInD V₂₋₁₀ | accuracy: 19 -> 46 | Helped |
| Use Monte Carlo inference instead of PAL | Llama3 · BLInD V₂₋₁₀ | accuracy: 40 -> 87 | Helped |
Surprising findings
- Adding Graph Generation or Number Extraction subtasks sometimes lowered accuracy, because subtask errors and longer outputs hurt performance.
- Llama3 with Graph Generation did not improve Monte Carlo, since its standalone subtask accuracy was below its Monte Carlo accuracy on larger networks.
Limitations
- Only binary variables and networks with at most 10 nodes, with arborescence structure restriction.
- Accuracy was evaluated on 900 randomly sampled instances; few-shot examples were hand-picked and tuned on a dev set.
- Evaluated only on prompting; no model fine-tuning was explored.
Experimental setup
Datasets: BLInD CLADDER (hard test split, adapted)
Benchmarks: BLInD CLADDER adapted
Metrics: accuracy (within ±0.01 of ground truth) subtask accuracy
Baselines: Basic QA prompting Chain of Thought PAL
Builds on: Chain-of-thought prompting Program-aided Language Models (PAL) ProbLog CLADDER pgmpy
Closest papers by meaning
- Does Reasoning Emerge? Examining the Probabilities of Causation in Large Language Models (NeurIPS 2024 · similarity 0.90)
- TokUR: Token-Level Uncertainty Estimation for Large Language Model Reasoning (ICLR 2026 · similarity 0.88)
- Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs (ICLR 2026 · similarity 0.88)
- Training Language Models to Reason Efficiently (NeurIPS 2025 · similarity 0.88)
- Doing Experiments and Revising Rules with Natural Language and Probabilistic Reasoning (NeurIPS 2024 · similarity 0.88)
- FLUE: Streamlined Uncertainty Estimation for Large Language Models (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.