Research map / Large language models / Language, safety and interpretability

FLUE: Streamlined Uncertainty Estimation for Large Language Models

AAAI 2025 · Large language models › Language, safety and interpretability · approach: Uncertainty quantification for LLMs · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FLUE, a single-forward-pass uncertainty estimator for open-ended QA that samples hidden states across similar LLM layers via Dirichlet weighting and maps token uncertainties to sequence uncertainty with a state transition model.

Problem

Uncertainty estimation for open-ended QA with LLMs is costly because sampling-based methods need multiple autoregressive inferences, and token-level methods miss semantic dependencies across sequences.

Approach

Exploit the high similarity of hidden states across consecutive middle-to-upper decoder layers to inject Dirichlet-distributed randomness into a single forward pass (MC-FLUE), approximating MC-Dropout token entropy, then learn a state-transition post-processor from token uncertainties to sequence uncertainty.

Key results

• On white-box sequence-level uncertainty (Table 1), FLUE reaches average AUROC 0.808 on TriviaQA-based evaluation and the highest AUROC on TriviaQA, NQ-Open, and SQuAD 2.0 among baselines, while Semantic Entropy has the best overall average (about 0.87 on bioasq/svamp).
• On SQuAD 2.0 with LLaMA 2 7B, FLUE takes 15 minutes at AUROC 0.837 versus 1h16min at 0.774 for Semantic Entropy and 1h47min at 0.726 for Length Normalization.
• In the black-box setting, FLUE attains higher AUROC and lower ECE than Confidence Elicitation across five datasets and five models.

Measured effects reported in the paper

Surprising findings

• Confidence Elicitation performs near chance (AUROC around 0.5) for several small models, indicating poorly calibrated verbalized confidence.
• Semantic Entropy outperforms FLUE on the SVAMP and BioASQ datasets, which the authors attribute to clustering being advantageous in specialized domains.

Limitations

• Proposition 1 relies on an assumed small conditional entropy at high layers and an approximate upper bound, which is only loosely justified.
• Performance is not uniformly best; Semantic Entropy wins on SVAMP and BioASQ, and FLUE is sensitive to layer selection and sample count.
• Layer-range selection requires a similarity analysis on a sampled subset of data per model.

Experimental setup

Datasets: SQuAD 2.0 SVAMP NQ-open TriviaQA BioASQ

Benchmarks: SQuAD 2.0 SVAMP NQ-open TriviaQA BioASQ

Metrics: AUROC ECE Pearson correlation Spearman correlation inference time

Baselines: Length Normalization Semantic Entropy P(True) Confidence Elicitation LLM Self-Expression Uncertainty

Builds on: MC-Dropout Length Normalization Semantic Entropy P(True)

Closest papers by meaning

• Efficient semantic uncertainty quantification in language models via diversity-steered sampling (NeurIPS 2025 · similarity 0.92)
• SPANUQ: Span-Level Uncertainty Quantification for Large Language Model Generation (NeurIPS 2026 · similarity 0.91)
• Inv-Entropy: A Fully Probabilistic Framework for Uncertainty Quantification in Language Models (NeurIPS 2025 · similarity 0.89)
• Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similarities (NeurIPS 2024 · similarity 0.89)
• Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs (ICLR 2026 · similarity 0.89)
• TokUR: Token-Level Uncertainty Estimation for Large Language Model Reasoning (ICLR 2026 · similarity 0.89)