Research map / Large language models / Language, safety and interpretability
BIRD: A Trustworthy Bayesian Inference Framework for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BIRD, which uses LLM abduction to build a Bayesian network over LLM-generated factors, fits its conditional probabilities to LLM coarse estimates, and deduces probabilities for new conditions via LLM entailment.
Problem
LLMs produce inaccurate and overconfident numerical probability estimates under incomplete information, which limits their use in large-scale probabilistic decision-making.
Approach
Approximate a Bayesian network whose variables are natural-language factors generated by the LLM (abduction), learn its per-factor conditional probabilities by fitting LLM coarse verbalized estimates over factor-value combinations using a logarithmic-opinion-pool formula with MSE and margin ranking losses, then map new contexts to factor values via LLM entailment and marginalize (deduction).
Key results
- On a 350-instance MTurk pairwise evaluation, BIRD with Llama-2 and Llama-3.1 reaches F1 of 0.59, over 30% above GPT-4 vanilla verbalization (0.289 average for GPT4 CoT).
- BIRD (Llama-3.1, optimized) beats CoT with self-consistency on the temporal dataset (74.3 vs 72.6) and the planning dataset (73.0 vs 71.5) but trails CoT on commonsense (92.3 vs 90.8 is a win; Llama-2 commonsense 89.0 vs 93.8 CoT).
- Using BIRD-estimated probabilities as soft supervision raised T5-large average accuracy from 67.9 (hard label) to 68.9 and PatternTime from 69.7 to 71.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BIRD optimized probabilities instead of direct LLM probability estimation | Llama2-70b Instruct · Commonsense pairwise probability preference | F1 Average: 0.311 -> 0.592 | Helped |
| Use BIRD optimized probabilities instead of direct LLM probability estimation | Llama3.1-70b Instruct · Commonsense pairwise probability preference | F1 Average: 0.303 -> 0.588 | Helped |
| Use BIRD optimized probabilities instead of CoT for decisions | Llama-2-70b-instruct · MATRES, PLASMA, and Commonsense | accuracy: MATRES 71.5 -> 73.9; PLASMA 76.8 -> 74.0; Commonsense 93.8 -> 89.0 | Mixed |
| Use BIRD optimized probabilities instead of CoT for decisions | Llama-3.1-70b-Instruct · MATRES, PLASMA, and Commonsense | accuracy: MATRES 72.6 -> 74.3; PLASMA 71.5 -> 73.0; Commonsense 90.8 -> 92.3 | Helped |
| Replace the 1/2 assumption with constrained optimization (ablation) | BIRD · Decision-making benchmarks | accuracy: Llama-2: MATRES 73.7 -> 73.9, PLASMA 72.8 -> 74.0, Commonsense 86.9 -> 89.0; Llama-3.1: MATRES 65.5 -> 74.3, PLASMA 71.3 | Helped |
| Replace the 1/n assumption with constrained optimization (ablation) | BIRD · Decision-making benchmarks | accuracy: Llama-2: MATRES 72.8 -> 73.9, PLASMA 72.3 -> 74.0, Commonsense 86.8 -> 89.0; Llama-3.1: MATRES 68.9 -> 74.3, PLASMA 66.5 | Helped |
| Replace fixed initial probabilities with constrained optimization (ablation) | BIRD · Decision-making benchmarks | accuracy: Llama-2: MATRES 73.4 -> 73.9, PLASMA 72.7 -> 74.0, Commonsense 87.5 -> 89.0; Llama-3.1: MATRES 65.5 -> 74.3 | Helped |
| Replace the 1/2 assumption with optimized probabilities (ablation) | BIRD with Llama3.1-70b Instruct · Commonsense pairwise probability preference | F1 Average: 0.480 -> 0.588 | Helped |
| Replace the 1/n assumption with optimized probabilities (ablation) | BIRD with Llama3.1-70b Instruct · Commonsense pairwise probability preference | F1 Average: 0.532 -> 0.588 | Helped |
| Replace fixed initial probabilities with optimized probabilities (ablation) | BIRD with Llama3.1-70b Instruct · Commonsense pairwise probability preference | F1 Average: 0.568 -> 0.588 | Helped |
| Use abductive sampling instead of direct factor generation (ablation) | BIRD with Llama-2 · MATRES, PLASMA, and Commonsense | accuracy: MATRES 70.9 -> 73.9; PLASMA 68.6 -> 74.0; Commonsense 84.2 -> 89. | Helped |
| Use abductive sampling instead of direct factor generation (ablation) | BIRD with Llama-3.1 · MATRES, PLASMA, and Commonsense | accuracy: MATRES 68.9 -> 74.3; PLASMA 69.0 -> 73.0; Commonsense 89.9 -> 89. | Helped |
Surprising findings
- Under the explicit comparison setting, which sees both contexts together, GPT-4 outperforms BIRD by only about 5% despite the setting being unfair to BIRD.
- Llama-2 BIRD underperforms CoT on the commonsense decision-making set (89.0 vs 93.8).
Limitations
- Factor generation can be incomplete; error analysis shows 56.7% of errors stem from factors not covering all elements of the condition.
- Argmax decision-making underperforms CoT on some benchmarks, and 'unknown' predictions occur when conditions map to no factor.
- Relies on the assumption that factors are conditionally independent and complete, which is only approximated.
Experimental setup
Datasets: CommonsenseQA-derived decision set (called CSQA-based commonsense set in paper) PlanBench-style planning set (PlaSMA-derived) temporal reasoning dataset (TRAM-style, 1000 instances) MATRES TRACIE
Benchmarks: commonsense decision-making set planning set temporal reasoning set MATRES TRACIE
Metrics: F1 accuracy
Baselines: Vanilla verbalization Logit CoT with self-consistency Explicit Comparison 1/2 assumption 1/N assumption Fixed initial prob
Builds on: chain-of-thought logarithmic opinion pool
Closest papers by meaning
- Large Language Bayes (NeurIPS 2025 · similarity 0.90)
- Large Language Models Must Be Taught to Know What They Don’t Know (NeurIPS 2024 · similarity 0.88)
- A Probabilistic Perspective on Unlearning and Alignment for Large Language Models (ICLR 2025 · similarity 0.88)
- Large Language Models to Enhance Bayesian Optimization (ICLR 2024 · similarity 0.88)
- OptScale: Probabilistic Optimality for Inference-time Scaling (AAAI 2026 · similarity 0.88)
- PlanU: Large Language Model Reasoning through Planning under Uncertainty (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.