AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

Matchability-Aware Conformal Prediction for Open-Ended Language Model Generation

Ilia Azizi

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Statistical frameworks for LLM evaluation · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CAELUM, a conformal prediction framework that builds semantic prediction sets for open-ended LLM answers while separating aleatoric and epistemic uncertainty.

Problem

Open-ended LLM question answering rarely comes with calibrated uncertainty, so answers lack reliable prediction sets.

Approach

Samples answers under prompt-ensembled clarifications, clusters them by semantic equivalence, and estimates aleatoric and epistemic components to build conformal prediction sets that account for matchability.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add fitted epistemic weight lambda* to the score instead of aleatoric-only weight (ablation)CAELUM vs Vanilla CP · six QA sets, 3 LLMsaverage set size: 2.21 vs 2.21 mean APSSNo clear effect
add lambda* weighting (ablation)CAELUM · Ministral AmbigQAaverage set size: 4.08 -> 3.90Improved
use epistemic-only weight (lambda=1e6) (ablation)CAELUM · Ministral AmbigInstaverage set size: 2.32 -> 2.99Hurt
replace conformal sets with self-consistency proxyCAELUM · six datasetscoverage: up to 6.5 points lower coverageHurt
interpret lambda* as intrinsic ambiguity measureCAELUM · dataset-level lambda*interpretabilityNo clear effect
use epistemic uncertainty instead of total entropy to rank support failuresU_epi vs U_tot · 17 non-degenerate cellsAUROC: +0.014 averageImproved
use R(x) for routing instead of randomR(x) · selective answering (TriviaQA)AUC: +0.040 to +0.061Mixed
restrict calibration to learned matchability stratumVanilla CP · 30 cellscoverage: 0.65 -> 0.74Improved
replace Good-Turing trigger with learned CAELUM detectorsupport failure detection · 29 cellsAUROC: 0.550 -> 0.780Improved
use larger generator (Qwen3-8B to 32B)Qwen3 · pooled heuristic match ratematch rate: 61.1% -> 66.0%Improved
scale up sampling or generator on PopQAlearned-stratum CP · PopQAcoverage checkNo clear effect

Experimental setup

Builds on: conformal prediction

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy