Research map / Large language models / Language, safety and interpretability
LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows LLM internal states encode truthfulness most strongly at exact answer tokens, that probes generalize poorly across tasks beyond logit baselines, that hidden states predict error types, and that probes can pick correct answers the model fails to generate.
Problem
Understanding where and how LLMs encode truthfulness and errors (hallucinations) in their internal representations, and whether these signals generalize across tasks and link to external behavior.
Approach
Probing classifiers on hidden states at the exact answer tokens of long-form generations substantially improve error detection over last-token probing and logit baselines; the analysis then tests cross-task generalization, error-type prediction from resampled behavior, and whether probes can select correct answers from sampled candidates.
Key results
- Exact answer token probing reaches AUC 0.85 on TriviaQA and 0.92 on Winobias and Math for Mistral-7b-instruct, outperforming logit baselines (best logit baseline 0.75 on TriviaQA).
- Probes trained on one dataset mostly generalize only to tasks with similar skills; after subtracting the Logits-min-exact baseline, generalization rarely exceeds logit-only performance.
- Probes select correct answers from 30 samples with large gains on error types C2, D, and E1 (30-40 points in some cases), indicating a gap between internal encoding and generated output.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Probe the last exact answer token instead of the last generated token (ablation) | Mistral-7B-Instruct error detector · TriviaQA | AUC: 0.71 ± 0.006 -> 0.85 ± 0.004 | Helped |
| Probe the last exact answer token instead of the last generated token (ablation) | Mistral-7B-Instruct error detector · Winobias | AUC: 0.82 ± 0.004 -> 0.92 ± 0.005 | Helped |
| Probe the last exact answer token instead of the last generated token (ablation) | Mistral-7B-Instruct error detector · Math | AUC: 0.74 ± 0.008 -> 0.92 ± 0.008 | Helped |
| Probe the last exact answer token instead of the last generated token (ablation) | Llama3-8B-Instruct error detector · TriviaQA | AUC: 0.81 ± 0.005 -> 0.83 ± 0.002 | Helped |
| Probe the last exact answer token instead of the last generated token (ablation) | Llama3-8B-Instruct error detector · Winobias | AUC: 0.86 ± 0.007 -> 0.93 ± 0.004 | Helped |
| Probe the last exact answer token instead of the last generated token (ablation) | Llama3-8B-Instruct error detector · Math | AUC: 0.82 ± 0.016 -> 0.95 ± 0.027 | Helped |
| Train the probe on one dataset and test it on another | Probing error detectors · Cross-dataset tasks | AUC beyond logit-based detection | No clear effect |
| Use the error-detection probe to select from 30 sampled answers | Mistral-7B-Instruct · TriviaQA, mostly incorrect (C2), two competing (D), and many answers (E1) | accuracy: 30 to 40 points | Helped |
| Use the error-detection probe to select from sampled answers | Mistral-7B-Instruct · TriviaQA, mostly correct (B2) | accuracy | No clear effect |
Surprising findings
- Probes trained on one task generalize to other tasks mostly only as much as logit-based baselines do, so apparent universal truthfulness encoding is largely explained by logits.
- The probe is most effective at picking correct answers when the LLM has no external preference for the correct answer, indicating the model encodes correct answers it fails to generate.
Limitations
- Requires white-box access to internal activations, limiting use to open-weight models.
- Focuses on QA tasks with clear gold labels; open-ended tasks left for future work.
- Exact answer extraction relies on heuristics and an instruct LLM, with extraction success rates around 95-99%.
Experimental setup
Datasets: TriviaQA HotpotQA HotpotQA with context Natural Questions with context Winobias Winogrande MNLI Math IMDB Movies
Benchmarks: TriviaQA Winobias Math HotpotQA HotpotQA-WC Winogrande NLI NQ-WC Movies IMDB
Metrics: AUC accuracy
Baselines: Logits-mean Logits-min P(True) Probas-min-exact Last generated token probe Before last generated token probe End of question probe
Builds on: probing classifiers P(True) logit-based uncertainty estimation
Closest papers by meaning
- In-Context Sharpness as Alerts: An Inner Representation Perspective for Hallucination Mitigation (ICML 2024 · similarity 0.91)
- FactTest: Factuality Testing in Large Language Models with Finite-Sample and Distribution-Free Guarantees (ICML 2025 · similarity 0.91)
- How Language Model Hallucinations Can Snowball (ICML 2024 · similarity 0.90)
- DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models (ICLR 2024 · similarity 0.90)
- LLM-Check: Investigating Detection of Hallucinations in Large Language Models (NeurIPS 2024 · similarity 0.90)
- Why and How LLMs Hallucinate: Connecting the Dots with Subsequence Associations (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.