Research map / Large language models / Language, safety and interpretability
Language Models with Conformal Factuality Guarantees
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes conformal factuality, which applies split conformal prediction to LM outputs by backing off unreliable sub-claims (via entailment sets) to give high-probability correctness guarantees for black-box LMs such as GPT-4.
Problem
LM outputs cannot be guaranteed factual; standard conformal prediction is intractable or yields unusable sets for open-ended text generation.
Approach
Define uncertainty sets implicitly as the entailment set of an LM output, so correctness equals coverage; build a back-off sequence by removing low-confidence sub-claims and pick the threshold with split conformal calibration, yielding a single natural-language output with a marginal factuality guarantee.
Key results
- Frequency scoring raises FActScore factuality from about 25% to about 80% while keeping roughly half of the sub-claims.
- On NQ, factuality rises by about 15% by removing about a quarter of sub-claims; on MATH, about 15% gain removing about 10%.
- Empirical factuality closely tracks the target 1-alpha across 1000 random calibration/test splits.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply conformal factuality back-off to GPT-4 outputs using frequency scoring | GPT-4 · FActScore | correctness: 30% -> 80% | Helped |
| Apply conformal factuality back-off to GPT-4 outputs using frequency scoring | GPT-4 · NaturalQuestions | correctness: 78% -> 93% | Helped |
| Apply conformal factuality back-off to GPT-4 outputs using frequency scoring | GPT-4 · MATH | correctness: 75% -> 95% | Helped |
| Use frequency scoring to rank sub-claims for removal (ablation) | Conformal factuality with GPT-4 · FActScore | correctness: ∼25% to ∼80% | Helped |
| Use frequency scoring to rank sub-claims for removal (ablation) | Conformal factuality with GPT-4 · NaturalQuestions | correctness: ∼15% | Helped |
| Use frequency scoring to rank sub-claims for removal (ablation) | Conformal factuality with GPT-4 · MATH | correctness: ∼15% | Helped |
| Allow partial factuality rather than requiring all accepted sub-claims to be factual (ablation) | Conformal factuality · FActScore, NaturalQuestions, and MATH | sub-claims removed | Mixed |
Surprising findings
- Base GPT-4 outputs on FActScore were factual for only about 25% of examples, showing aggressive hallucination.
- Partial-factuality relaxation yielded only small gains in utility, according to the authors.
Limitations
- Guarantees are marginal over inputs and calibration draw, not conditional per input or per user.
- Guarantees assume exchangeability; distribution shift can break coverage.
- Evaluation uses only 50 annotated examples per dataset.
Experimental setup
Datasets: FActScore Natural Questions MATH
Benchmarks: FActScore Natural Questions MATH
Metrics: factuality rate percent of sub-claims removed empirical coverage vs target alpha
Baselines: random scoring ordinal scoring GPT-4 confidence scoring oracle scoring base GPT-4
Builds on: split conformal prediction SelfCheckGPT self-consistency
Closest papers by meaning
- Large language model validity via enhanced conformal prediction methods (NeurIPS 2024 · similarity 0.91)
- Conformal Linguistic Calibration: Trading-off between Factuality and Specificity (NeurIPS 2025 · similarity 0.91)
- Conformal Language Model Reasoning with Coherent Factuality (ICLR 2025 · similarity 0.90)
- Inference-Time Conformal Reasoning with Valid Factuality Control for Large Language Models (ICML 2026 · similarity 0.90)
- Conformal Language Modeling (ICLR 2024 · similarity 0.90)
- Distinguishing the Knowable from the Unknowable with Language Models (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.