Research map / Large language models / Retrieval and knowledge
Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses Gemma Scope and LlamaScope sparse autoencoders to find entity recognition latents in base models that detect known vs unknown entities, and shows steering them causally controls knowledge refusal and hallucination in chat models.
Problem
Hallucinations in LLMs occur when models generate information for entities they do not know, and the mechanisms determining whether a model hallucinates or refuses are poorly understood.
Approach
Sparse autoencoder latents on the final token of an entity in the base model separate known from unknown entities across entity types; steering these latents (added to the residual stream) causally flips knowledge refusal in the chat model and modulates attention from attribute-extraction heads to the entity.
Key results
- Steering with the unknown entity latent induces near 100% refusal across entity types in Gemma 2 2B IT, and weight orthogonalization of that direction sharply reduces refusals.
- A latent in the Gemma 2 2B IT residual stream at the model token predicts incorrect answers with 73.2 AUROC and F1 of 72.
- Known entity latents activate on 6% of post-cutoff song titles in Gemma 2 2B versus unknown latents at 53%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Steer with the unknown entity recognition latent | Gemma 2 2B chat model · Questions about unknown entities | Knowledge refusal rate | Helped |
| Steer with the known entity recognition latent | Gemma 2 2B chat model · Questions about unknown entities | Knowledge refusal rate | Hurt |
| Orthogonalize model weights with respect to the unknown entity direction | Gemma 2 2B chat model · Questions about unknown entities | Knowledge refusal rate | Hurt |
| Steer with the known entity latent on an unknown entity | Gemma 2 2B chat model · Question about Wilson Brown | Hallucination tendency | Helped |
| Steer with the unknown entity recognition latent | Gemma 2 2B · Known-entity prompts | Attention to entity tokens | Hurt |
| Steer with the known entity recognition latent | Gemma 2 2B · Unknown-entity prompts | Attention to entity tokens | Helped |
| Steer with the unknown entity recognition latent | Gemma 2 2B, Gemma 2 9B, and Llama 3.1 8B · Entity prompts | Attention to entity tokens versus random SAE latents: 9/10; 1/10; 10/10 | Mixed |
| Steer with the known entity recognition latent | Gemma 2 2B, Gemma 2 9B, and Llama 3.1 8B · Entity prompts | Attention to entity tokens versus random SAE latents: 10/10; 10/10; 7/10 | Helped |
| Use the entity recognition latent selected by minimum separation across entity types | Gemma 2 models and Llama 3.1 8B · Questions about entities | Recognition of known versus unknown entities | Helped |
| Use the unknown-entity uncertainty latent activation as a classifier | Gemma 2 2B IT · Correct versus mistaken answers | AUROC: 73.2 AUROC | Helped |
| Use entity recognition latents to steer explicit knowledge judgments | Gemma 2 chat models · “Are you sure you know” prompts | Yes-versus-No logit difference | Helped |
| Use the unknown entity latent rather than a random SAE latent for steering | Gemma 2 2B, Gemma 2 9B, and Llama 3.1 8B · FineWeb tokens | Correlation between latent activation and token prediction probability | No clear effect |
Surprising findings
- SAEs trained on the base model yield directions that causally control refusal in the chat model, suggesting chat finetuning repurposes existing mechanisms.
- Token likelihood is not a major confound: correlations between latent activations and next-token probability are negligible.
Limitations
- Known/unknown labels come from fuzzy string matching against model answers, which can mislabel guessed or unrecalled attributes.
- Effects are less pronounced in Gemma 2 9B and Llama 3.1 8B than in Gemma 2 2B.
- Steering coefficients are tuned on a validation set, and the uncertainty latent analysis covers only a few layers with available SAEs.
Experimental setup
Datasets: Wikidata entities (players, movies, cities, songs) FineWeb The Pile Neuronpedia explanations
Benchmarks: Wikidata entity attribute prompts (known/unknown split)
Metrics: latent separation score refusal count out of 100 attention score change to entity tokens logit difference Yes vs No AUROC F1
Baselines: random SAE latent steering random vector steering Orthogonalized model
Builds on: Gemma Scope Refusal is mediated by a single direction activation patching JumpReLU SAE LlamaScope
Closest papers by meaning
- In-Context Sharpness as Alerts: An Inner Representation Perspective for Hallucination Mitigation (ICML 2024 · similarity 0.89)
- ReDeEP: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretability (ICLR 2025 · similarity 0.87)
- DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models (ICLR 2024 · similarity 0.87)
- Auditing Meta-Cognitive Hallucinations in Reasoning Large Language Models (NeurIPS 2025 · similarity 0.87)
- How Language Model Hallucinations Can Snowball (ICML 2024 · similarity 0.86)
- FactTest: Factuality Testing in Large Language Models with Finite-Sample and Distribution-Free Guarantees (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.