Research map / Large language models / Language, safety and interpretability

Distinguishing the Knowable from the Unknowable with Language Models

ICML 2024 · Large language models › Language, safety and interpretability · approach: Statistical frameworks for LLM evaluation · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that linear probes on a small LLM's activations can predict when a much larger proxy LLM will be confident (epistemic vs aleatoric token uncertainty), transfer across text domains, and an unsupervised in-context learning test (ICLT) gives non-trivial separation.

Problem

Identifying epistemic uncertainty (lack of knowledge) rather than aleatoric uncertainty (inherent entropy) in free-form LLM token predictions, without ground-truth distributions.

Approach

Use a much larger model's predictive entropy as a proxy for the ground truth to label tokens as epistemic or aleatoric, train small probes on the small model's frozen embeddings to predict these labels, and propose an unsupervised In-Context Learning Test that measures how much the small model copies a repeated hint in its context.

Key results

• Linear probes on LLaMA 7B activations predict LLaMA 30B near-zero-entropy tokens with AUC 0.94 on Wikipedia (in-band [2,3)), versus 0.54 for the best entropy threshold baseline.
• Probes trained on Wikipedia transfer out of distribution to Pile StackExchange with AUC 0.88 for LLaMA 7B/30B and to Code with AUC 0.82.
• The unsupervised ICLT achieves AUC 0.71 (accuracy 0.67) for LLaMA 7B/30B in band [2,3), compared with 0.56 for the baseline; it fails for Pythia (AUC about 0.54).

Measured effects reported in the paper

Surprising findings

• ICLT fails for Pythia models (AUC around 0.53-0.54), and the authors link this to the document separator token.
• Using a larger 'large' model (30B vs 65B LLaMA) did not substantially change the difficulty of the task.

Limitations

• Large proxy models still exhibit epistemic uncertainty, introducing label noise.
• The binary higher-entropy class is a catch-all that mixes aleatoric and epistemic uncertainty on both models.
• Performance is weaker on Pythia and on the small EuroParl set; ICLT is only non-trivially effective for LLaMA.

Experimental setup

Datasets: Wikipedia articles (post-cutoff, June 2023) The Pile evaluation and test sets (Code, StackExchange, EuroParl)

Benchmarks: Wikipedia held-out test set The Pile Code The Pile EuroParl The Pile StackExchange

Metrics: AUROC accuracy

Baselines: Best entropy threshold (BET) BET-FT Small model entropy (SME) Prediction from initial embeddings (PIE)

Builds on: Epistemic Neural Networks Contrast-Consistent Search

Closest papers by meaning

• Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similarities (NeurIPS 2024 · similarity 0.91)
• Decomposing Uncertainty for Large Language Models through Input Clarification Ensembling (ICML 2024 · similarity 0.91)
• Efficient semantic uncertainty quantification in language models via diversity-steered sampling (NeurIPS 2025 · similarity 0.90)
• Improving Uncertainty Estimation through Semantically Diverse Language Generation (ICLR 2025 · similarity 0.90)
• To Believe or Not to Believe Your LLM: Iterative Prompting for Estimating Epistemic Uncertainty (NeurIPS 2024 · similarity 0.90)
• Forking Paths in Neural Text Generation (ICLR 2025 · similarity 0.89)