Research map / Large language models / Language, safety and interpretability
Matchability-Aware Conformal Prediction for Open-Ended Language Model Generation
Ilia Azizi
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CAELUM, a conformal prediction framework that builds semantic prediction sets for open-ended LLM answers while separating aleatoric and epistemic uncertainty.
Problem
Open-ended LLM question answering rarely comes with calibrated uncertainty, so answers lack reliable prediction sets.
Approach
Samples answers under prompt-ensembled clarifications, clusters them by semantic equivalence, and estimates aleatoric and epistemic components to build conformal prediction sets that account for matchability.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add fitted epistemic weight lambda* to the score instead of aleatoric-only weight (ablation) | CAELUM vs Vanilla CP · six QA sets, 3 LLMs | average set size: 2.21 vs 2.21 mean APSS | No clear effect |
| add lambda* weighting (ablation) | CAELUM · Ministral AmbigQA | average set size: 4.08 -> 3.90 | Improved |
| use epistemic-only weight (lambda=1e6) (ablation) | CAELUM · Ministral AmbigInst | average set size: 2.32 -> 2.99 | Hurt |
| replace conformal sets with self-consistency proxy | CAELUM · six datasets | coverage: up to 6.5 points lower coverage | Hurt |
| interpret lambda* as intrinsic ambiguity measure | CAELUM · dataset-level lambda* | interpretability | No clear effect |
| use epistemic uncertainty instead of total entropy to rank support failures | U_epi vs U_tot · 17 non-degenerate cells | AUROC: +0.014 average | Improved |
| use R(x) for routing instead of random | R(x) · selective answering (TriviaQA) | AUC: +0.040 to +0.061 | Mixed |
| restrict calibration to learned matchability stratum | Vanilla CP · 30 cells | coverage: 0.65 -> 0.74 | Improved |
| replace Good-Turing trigger with learned CAELUM detector | support failure detection · 29 cells | AUROC: 0.550 -> 0.780 | Improved |
| use larger generator (Qwen3-8B to 32B) | Qwen3 · pooled heuristic match rate | match rate: 61.1% -> 66.0% | Improved |
| scale up sampling or generator on PopQA | learned-stratum CP · PopQA | coverage check | No clear effect |
Experimental setup
Builds on: conformal prediction
Closest papers by meaning
- Large language model validity via enhanced conformal prediction methods (NeurIPS 2024 · similarity 0.90)
- Geometry-Calibrated Conformal Abstention for Language Models (NeurIPS 2026 · similarity 0.90)
- Conformal Language Modeling (ICLR 2024 · similarity 0.89)
- Conformal Prediction Beyond the Seen: A Missing Mass Perspective for Uncertainty Quantification in Generative Models (NeurIPS 2025 · similarity 0.89)
- Distinguishing the Knowable from the Unknowable with Language Models (ICML 2024 · similarity 0.88)
- Conformal Linguistic Calibration: Trading-off between Factuality and Specificity (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
