Research map / Large language models / Language, safety and interpretability
Conformal Linguistic Calibration: Trading-off between Factuality and Specificity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Conformal Linguistic Calibration, which reframes linguistic calibration as answer set prediction by rewriting answers into less specific claims over nested answer clusters, with LTT-based factuality guarantees, and distills the rewriter into Llama-3-8B.
Problem
LLM answers are often overconfident and unreliable; abstention withholds information, while linguistic hedging is hard to use downstream. The paper seeks a way to trade off factuality and specificity with probabilistic factuality guarantees.
Approach
Treat linguistic calibration as soft abstention: sample answers, cluster them into nested answer sets ordered by multiplicity, rewrite each nested set into a vaguer claim that describes the set (belief probing), and select the level with Learn-Then-Test to control factual error.
Key results
- On SimpleQA, a 7B model with CLC outperforms GPT-4o in factuality by answering less specifically (per the paper's claim).
- CLC answers all questions while improving factuality, whereas abstention gains are limited on challenging SimpleQA.
- CPMI informativeness decreases steadily as the multiplicity threshold rises, showing an explicit factuality-informativeness trade-off; naive rewriting fails to do so.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| rewrite responses into less specific claims using CLC instead of abstaining | Llama3-8B-Instruct · SimpleQA | factuality | Helped |
| rewrite responses into less specific claims using CLC instead of abstaining | Llama3-8B-Instruct · Natural Questions dev | factuality | Helped |
| increase the answer multiplicity threshold | CLC · SimpleQA | factuality | Helped |
| increase the answer multiplicity threshold | CLC · Natural Questions | factuality | Helped |
| increase the answer multiplicity threshold | CLC · SimpleQA | CPMI | Hurt |
| increase the answer multiplicity threshold | CLC · Natural Questions | CPMI | Hurt |
| use naive direct rewriting instead of CLC's calibrated rewriting (ablation) | Llama-3-7b-Instruct · SimpleQA and Natural Questions | informativeness trade-off | Hurt |
| use naive direct rewriting instead of CLC's calibrated rewriting (ablation) | Llama-3-7b-Instruct · SimpleQA and Natural Questions | informativeness | Hurt |
| fine-tune Llama-3-8B-Instruct on CLC rewriting data | claim rewriter · FActScore Core dev | FActScore | Helped |
| apply the out-of-the-box prompting baseline instead of the tuned rewriter (ablation) | Llama-3-8B-Instruct claim rewriter · FActScore Core dev | instruction adherence | Hurt |
| frame answer clusters as an error-prone agent's belief rather than objective facts (ablation) | CLC belief summarization · claim rewriting | faithfulness of belief generation | Helped |
| use majority-vote answer multiplicity to order the nested clusters | CLC · SimpleQA and Natural Questions | factuality | Helped |
Surprising findings
- Naive prompted rewriting does not yield a fine-grained trade-off and tends to degenerate to low-informativeness rewrites.
Limitations
- LTT does not guarantee monotonicity of rewrites across nested levels; the summaries are only checked by a small manual verification (11 samples per question type).
- Factuality evaluation relies on an LLM judge with modified prompts that do not penalize generality.
Experimental setup
Datasets: SimpleQA Natural Questions FActScore Core biographies Wikipedia dump
Benchmarks: SimpleQA Natural Questions FActScore
Metrics: factuality accuracy conformal coverage guarantee CPMI informativeness FActScore
Baselines: abstention baseline naive rewriting baseline Llama-3-8B-Instruct base rewriter SFT rewriter ORPO rewriter
Builds on: Learn-Then-Test semantic uncertainty possible world semantics FActScore DPO
Closest papers by meaning
- Language Models with Conformal Factuality Guarantees (ICML 2024 · similarity 0.91)
- Geometry-Calibrated Conformal Abstention for Language Models (NeurIPS 2026 · similarity 0.90)
- QA-Calibration of Language Model Confidence Scores (ICLR 2025 · similarity 0.89)
- A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering (NeurIPS 2026 · similarity 0.89)
- Large language model validity via enhanced conformal prediction methods (NeurIPS 2024 · similarity 0.89)
- Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.