Research map / Large language models / Language, safety and interpretability
SelfIE: Self-Interpretation of Large Language Model Embeddings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SelfIE, which lets an LLM interpret its own hidden embeddings in natural language by inserting them into a second forward pass, and uses those interpretations for supervised and reinforcement-based control of model behavior.
Problem
LLM internal reasoning is opaque; existing probing methods need training data and cover only small closed concept sets, while logit-lens style decoders give only short token-level descriptions.
Approach
Replace placeholder token embeddings in an interpretation prompt with a hidden embedding from the original forward pass, so the same LLM decodes open-world natural-language descriptions of that embedding without any training; the interpretations then localize single-layer edits (supervised control) and enable RLHF-style reward on embeddings (reinforcement control).
Key results
- SelfIE interpretations recover 60%-80% of TextWorld entity-state information, performing similarly to a 100-sample linear probe.
- Supervised control reaches a 59.77% harmonic mean on CounterFact editing, the best among FT, RepE, and ROME, with 36.12% paraphrase accuracy.
- Reinforcement control reduces prompt-injection harmful response rate from 89.06% to 4.4% while keeping 95.85% fact accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use supervised control to override the prompted human-priority preference with equality | LLaMA-2-70B-Chat · Ethical preference scenarios | Prioritize equal: 0% -> 96% | Helped |
| Use reinforcement control to erase harmful knowledge elicited by a single non-injection prompt | LLaMA-2-70B-Chat · 388 harmful activities with prompt injection | Responses containing harmful information: 89.06% -> 4.4% | Helped |
| Use reinforcement control to erase harmful knowledge | LLaMA-2-70B-Chat · Counterfact | Irrelevant fact accuracy: 100% -> 95.85% | Hurt |
| Use SelfIE-based supervised control for fact editing | LLaMA-2-70B-Chat · Counterfact | Harmonic mean: 56.92% -> 59.77% | Helped |
| Use SelfIE-based supervised control for fact editing | LLaMA-2-70B-Chat · Counterfact | Paraphrase: 33.29% -> 36.12% | Helped |
| Use SelfIE-based supervised control for fact editing | LLaMA-2-70B-Chat · Counterfact | Specificity: 94.94% -> 90.43% | Hurt |
| Use SelfIE to elicit entity states without training | LLaMA-2-70B-Chat · TextWorld | Accuracy: 60%-80% | Helped |
| Use a 7B model instead of a 70B model (ablation) | SelfIE · TextWorld | Interpretation responses not following instruction: 3.84% -> 27.17% | Hurt |
| Use a 13B model instead of a 70B model (ablation) | SelfIE · TextWorld | Interpretation responses not following instruction: 3.84% -> 17.68% | Hurt |
| Insert the interpreted embedding at later layers (k >= 20) (ablation) | SelfIE · TextWorld | Accuracy | Hurt |
| Condition on successfully following the choice instruction for smaller models (ablation) | SelfIE with 7B or 13B LLaMA-2-Chat · TextWorld | Accuracy | No clear effect |
Surprising findings
- Smaller LLaMA-2 chat models (7B, 13B) perform worse mainly because they fail to follow the interpretation instruction (27.17% and 17.68% failure) rather than failing to recover embedding content.
Limitations
- Smaller models often fail to follow the interpretation instruction, degrading interpretation quality.
- Accuracy degrades for later insertion layers, so interpretation works best for early insertion layers (k < 20).
- Supervised control has lower efficacy (58.40%) than ROME (50.95%) and FT (82.12%) trade-offs; FT and RepE underperform on paraphrase and specificity.
Experimental setup
Datasets: TextWorld CounterFact WikiText ethical preference dataset (synthetic, 101 scenarios) harmful behaviors dataset (388 behaviors)
Benchmarks: TextWorld entity state elicitation CounterFact ethical preference override harmful knowledge erasure with prompt injection
Metrics: accuracy harmonic mean efficacy paraphrase specificity harmful response rate fact accuracy
Baselines: FT ROME RepE linear probe MEND TunedLens
Builds on: logit lens ROME RepE RLHF
Closest papers by meaning
- Large Language Models Cannot Self-Correct Reasoning Yet (ICLR 2024 · similarity 0.87)
- SELF-DISCOVER: Large Language Models Self-Compose Reasoning Structures (NeurIPS 2024 · similarity 0.87)
- Soft Reasoning: Navigating Solution Spaces in Large Language Models through Controlled Embedding Exploration (ICML 2025 · similarity 0.87)
- Training Language Models to Explain Their Own Computations (NeurIPS 2026 · similarity 0.87)
- SelfJudge: Faster Speculative Decoding via Self-Supervised Judge Verification (ICML 2026 · similarity 0.87)
- Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.