Research map / Large language models / Language, safety and interpretability
Monitoring Latent World States in Language Models with Propositional Probes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces propositional probes that decode latent world-state propositions from Tulu-2-13b activations by composing linear domain probes with a Hessian-derived binding subspace, and shows they stay faithful under prompt injection, backdoor, and gender-bias settings where prompting fails.
Problem
Language models can give responses unfaithful to their input context due to bias, sycophancy, backdoors, or injections; it is unclear whether their internal representations of the input world state remain faithful, and existing probes do not decode relational structure (who-has-which-attribute) from
Approach
Compose linear lexical (domain) probes with a binding similarity metric defined on a low-rank binding subspace obtained from the Hessian of a binding-strength function, so propositions are decoded compositionally rather than enumerated; probes trained only on templated synthetic text generalize to stories and Spanish.
Key results
- Propositional probes reach Jaccard 0.99/0.90/0.78 on SYNTH/PARA/TRANS, comparable to prompting (1.00/0.98/0.78), but EM drops to 0.26 on TRANS vs 0.40 for prompting.
- Under prompt injection on TRANS, probes keep Jaccard 0.76 while prompting falls to 0.51; on the backdoored model, probes get Jaccard 0.68 while prompting gets 0.00.
- Ablating the Hessian binding metric with DAS-50 subspace drops SYNTH EM from 0.97 to 0.00, showing the Hessian subspace is key.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use propositional probes instead of prompting | Tulu-2-13B · synth, para, and trans | Jaccard: synth: 1.00 -> 0.99; para: 0.98 -> 0.90; trans: 0.78 -> 0.78 | No clear effect |
| use propositional probes instead of prompting | Tulu-2-13B · synth, para, and trans | EM: synth: 1.00 -> 0.97; para: 0.93 -> 0.55; trans: 0.40 -> 0.26 | Hurt |
| use propositional probes instead of prompting | Tulu-2-13B · synth (P), para (P), trans (P), and trans (FT) | Jaccard: synth (P): 0.49 -> 0.99; para (P): 0.48 -> 0.90; trans (P): 0.51 -> 0.76; trans (FT): 0.00 -> 0.68 | Helped |
| use propositional probes instead of prompting | Tulu-2-13B · synth (P), para (P), trans (P), and trans (FT) | EM: synth (P): 0.07 -> 0.98; para (P): 0.04 -> 0.55; trans (P): 0.06 -> 0.24; trans (FT): 0.00 -> 0.09 | Helped |
| replace the Hessian binding metric with a 50-dim DAS subspace (ablation) | Prop. Probes · synth | EM: 0.97 -> 0.00 | Hurt |
| replace the Hessian binding metric with a 50-dim DAS subspace (ablation) | Prop. Probes · para | EM: 0.55 -> 0.01 | Hurt |
| replace the Hessian binding metric with a 50-dim DAS subspace (ablation) | Prop. Probes · trans | EM: 0.26 -> 0.00 | Hurt |
| replace the Hessian binding metric with a 1-dim DAS subspace (ablation) | Prop. Probes · synth, para, and trans | EM: synth: 0.97 -> 0.00; para: 0.55 -> 0.00; trans: 0.26 -> 0.00 | Hurt |
| replace the Hessian binding metric with a random 50-dim subspace (ablation) | Prop. Probes · synth, para, and trans | EM: synth: 0.97 -> 0.00; para: 0.55 -> 0.00; trans: 0.26 -> 0.00 | Hurt |
| constrain propositions to bind to unique entities (ablation) | Prop. Probes · nested | EM: 44% -> 76% | Helped |
Surprising findings
- Prompting on the backdoored model scores exact-match 0.00 and Jaccard 0.00, while the probes still decode mostly correct propositions from the same model's activations.
- Probes are robust to most entity-attribute orderings but reach only 44% EM on the nested ordering, with errors partly reflecting order bias captured by the binding subspace.
Limitations
- Closed world of 60 names, 16 countries, 41 foods, 14 occupations, and only two people per context, so scaling to larger worlds is untested.
- Three-entity contexts show the binding metric does not clearly discriminate second and third entities, indicating the subspace may be noisy.
- EM on the translated Spanish set is substantially lower than prompting (0.26 vs 0.40).
Experimental setup
Datasets: SYNTH (templated synthetic) PARA (GPT-3.5-turbo story rewrites) TRANS (GPT-3.5-turbo Spanish translations) Winobias occupations
Benchmarks: SYNTH PARA TRANS prompt-injected variants backdoored Spanish TRANS gender bias templated contexts
Metrics: exact-match accuracy Jaccard index calibrated accuracy interchange intervention accuracy
Baselines: prompting skyline Distributed Alignment Search (DAS-50, DAS-1) random subspace
Builds on: Distributed Alignment Search difference-in-means probes Grad-CAM binding ID mechanism (Feng & Steinhardt)
Closest papers by meaning
- Training Language Models to Explain Their Own Computations (NeurIPS 2026 · similarity 0.88)
- Walk the Talk? Measuring the Faithfulness of Large Language Model Explanations (ICLR 2025 · similarity 0.87)
- Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement (ICLR 2024 · similarity 0.87)
- Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models (ICML 2024 · similarity 0.87)
- Do LLMs Build World Representations? Probing Through the Lens of State Abstraction (NeurIPS 2024 · similarity 0.87)
- Evaluating the World Model Implicit in a Generative Model (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.