Research map / Large language models / Language, safety and interpretability

Latent Introspection: Models Can Detect Prior Concept Injections

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Reasoning mechanisms and failure modes

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that Qwen2.5-Coder-32B-Instruct can detect and identify concepts injected into its earlier KV cache, with detection hidden in sampled outputs but visible via logit lens in middle layers and boosted by prompting.

Problem

Whether open-weight language models can introspect on transient internal states, i.e., detect that concept vectors were injected into their earlier context, and whether behavioral sampling underestimates this capacity.

Approach

Inject steering vectors only during KV-cache generation of a first turn, remove steering, then query the model about injection; analyze output logits, logit lens across layers, and mutual information over concept identification, with a 4x4 factorial of prompt framings and info documents.

Key results

• Accurate Mechanism framing with No Document yields near-chance balanced accuracy (50.1%), while Vague Mechanism with Matched Lipsum Filler reaches 84% accuracy.
• Pro-introspection document raises sensitivity from 0.3% to 39.9% with only a 0.6% increase in false positives under Accurate Mechanism framing.
• Concept identification reaches 1.36 bits mutual information (43% of the 3.17-bit maximum), correlating with detection sensitivity across 16 conditions (r = 0.68, p = 0.004).

Surprising findings

• Vague mechanism framing outperforms mechanistically accurate descriptions, and wrong (finetuning) mechanism performs similarly to accurate.
• Llama-70B shows an inverted info-document effect, with the introspection document decreasing accuracy.

Limitations

• Results are highly prompt-dependent and the reasons for prompt sensitivity are not understood.
• No mechanistic circuit identification or causal intervention on the signal.
• Replications on larger models are single-seed and limited in scope; the emergent misalignment extension is preliminary.

Experimental setup

Datasets: contrastive concept prompts control question sets (always-yes, always-no, varied-baseline, confusing)

Benchmarks: concept identification over 9 concepts introspection detection with control questions

Metrics: P(yes) shift balanced accuracy mutual information (bits) Pearson correlation logit lens probability

Baselines: Claude Opus 4 Claude Opus 4.1

Builds on: concept injection (Lindsey) repeng logit lens

Closest papers by meaning

• Imagination Helps Visual Reasoning, But Not Yet in Latent Space (ICML 2026 · similarity 0.85)
• Looking Inward: Language Models Can Learn About Themselves by Introspection (ICLR 2025 · similarity 0.85)
• ActivationReasoning: Logical Reasoning in Latent Activation Spaces (ICLR 2026 · similarity 0.85)
• Predictive Concept Decoders: Training Scalable End-to-End Interpretability Assistants (NeurIPS 2026 · similarity 0.85)
• A Language Model’s Guide Through Latent Space (ICML 2024 · similarity 0.84)
• You Can’t Have It Both Ways: Concept Entanglement Limits Diffusion Model Unlearning (NeurIPS 2026 · similarity 0.84)