Research map / Large language models / Language, safety and interpretability

SPANUQ: Span-Level Uncertainty Quantification for Large Language Model Generation

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Statistical frameworks for LLM evaluation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Formalizes span-level uncertainty estimation (SLUE) and introduces SpanUQ, a ~25M-parameter DETR-style probe over frozen LLM hidden states that distills multi-sample consistency into one forward pass, plus the SpanUQ-Bench dataset.

Problem

Uncertainty estimation for LLMs is available only at token level (semantically incomplete) or sequence level (cannot localize errors), and sampling-based methods are expensive; span-level uncertainty for claims is missing.

Approach

A lightweight probe on frozen LLM hidden states uses DETR-style learnable span queries to jointly detect semantically coherent spans and predict each span's uncertainty as a Mixture of Beta, trained with Beta NLL, contrastive ranking, and consistency losses; labels come from cross-sample contradiction rates of 20 sampled responses.

Key results

• SpanUQ achieves span-level AUROC 0.939 on Qwen3-14B vs 0.881 for the strongest probe baseline (MLP Probe with oracle boundaries), with AUROC 0.908-0.944 across five backbones.
• The DETR-based span detector reaches 0.910 F1, 39.4% above the best heuristic (sliding window 20, F1 0.653).
• On human-annotated labels, rejecting the top 20% uncertain spans reduces factual error from 18.6% to 10.5%.

Surprising findings

• Sampling-based INSIDE and KLE produce negatively correlated span/sequence estimates (Spearman -0.331 and -0.145) when aggregated to sequence level.
• P(True) performs near chance (AUROC 0.526) because the LLM predicts 'True' for 93.8% of claims.

Limitations

• Requires white-box access to hidden states and cannot capture uncertainty from the generation process itself.
• Labels measure cross-sample consistency rather than factuality; 22.3% of high-uncertainty spans are factually correct and confident shared hallucinations are missed.
• Multi-sample label construction is compute-intensive and evaluation is limited to English claim uncertainty.

Experimental setup

Datasets: Natural Questions TriviaQA ELI5 FActScore biography FELM TruthfulQA HaluEval

Benchmarks: SpanUQ-Bench

Metrics: AUROC MAE Spearman ECE Detection F1 AURC PRR

Baselines: Token Entropy MLP Probe Verbalized Confidence SelfCheckGPT-NLI P(True) FActScore HaloScope HaMI Semantic Entropy INSIDE KLE

Builds on: DETR Semantic Entropy

Closest papers by meaning

• FLUE: Streamlined Uncertainty Estimation for Large Language Models (AAAI 2025 · similarity 0.91)
• Conformal Prediction with Paraphrase-Aware Scoring for LLM Uncertainty Quantification (NeurIPS 2026 · similarity 0.90)
• Graph-based Uncertainty Metrics for Long-form Language Model Generations (NeurIPS 2024 · similarity 0.90)
• Localizing Input Uncertainty Quantification for Large Language Models via Shapley Values (NeurIPS 2026 · similarity 0.89)
• Efficient semantic uncertainty quantification in language models via diversity-steered sampling (NeurIPS 2025 · similarity 0.89)
• Inv-Entropy: A Fully Probabilistic Framework for Uncertainty Quantification in Language Models (NeurIPS 2025 · similarity 0.89)