Research map / Architectures and efficiency / Attention and transformers
Emergence of Linear Truth Encodings in Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the Truth Co-occurrence Hypothesis and a one-layer transformer toy model showing that linear truth subspaces can emerge from training on data where true/false statements co-occur, with a two-phase (memorize then encode truth) dynamic.
Problem
Why and how linear subspaces separating true from false statements emerge in language models during training, which remains mechanistically unexplained.
Approach
Truth Co-occurrence Hypothesis: true statements co-occur with true statements and false with false, so inferring a latent truth bit lowers LM loss; a minimal attention+layer-norm transformer trained on such synthetic data learns a linear truth direction via key-value memorization followed by normalization-driven sharpening.
Key results
- Two-phase dynamics in a 1-layer model with rho=0.99: memorization within ~1000 batches, then linear truth encoding emerges abruptly after ~7,500 batches.
- In LLaMA3-8B, preceding a sentence with two false sentences raises negative log-likelihood of the correct attribute by 1.52 (about 4.55x lower probability) versus one true sentence.
- A linear probe reaches over 95% accuracy on middle and last layers of LLaMA3-8B separating true from counterfactual instances.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add normalization after attention (ablation) | Toy one-layer transformer · Synthetic true and false sequences | linear truth separability | Helped |
| Use two false preceding sentences instead of one true preceding sentence | Llama3-8B · CounterFact SpeaksLanguage | negative log likelihood: 1.52 | Hurt |
| Steer representations toward the truth direction with α=3.0 at layer l=11 | Llama3-8B · CounterFact SpeaksLanguage with false context | correct-attribute probability | Helped |
| Increase the true-attribute rate (ablation) | Synthetic transformer · Synthetic data, up to ρ=0.999 | time to linear truth separability | Hurt |
| Set the true-attribute rate to ρ=1.0 (ablation) | Fully trained synthetic transformer · Synthetic data | linear truth separability | Hurt |
| Train the key-query matrix when ρ=1 (ablation) | Synthetic transformer · False sequences | handling false sequences | Hurt |
| Train input embeddings instead of freezing them (ablation) | Dense-embedding transformer · Synthetic memorization and truth encoding | similarity to the idealized OV pattern | Hurt |
| Use a 3-layer model instead of a 1-layer model (ablation) | Synthetic transformer · Synthetic truth encoding | truth encoding location: 4/5 random initializations | Mixed |
| Train with frozen dense embeddings and freely learned KV matrices (ablation) | Synthetic transformer · Synthetic truth encoding | similarity to the idealized mechanism | Helped |
| Use a model dimension of 32 (ablation) | Synthetic transformer · Synthetic truth separability at the second-subject token | linear separability | Hurt |
Surprising findings
- Truth linear encoding emerges even when training only on true sequences (rho=1) in the toy model, generalizing out of distribution to false sequences, though it fails in the fully trainable model at rho=1.0.
Limitations
- Toy model uses a single latent relation and uniform corruption, far simpler than natural text.
- Results on pretrained LLMs are preliminary and the mechanism may not be the one used by real LMs.
- Fully trainable model at rho=1.0 does not show the emergence seen in the toy model.
Experimental setup
Datasets: MAVEN-FACT CounterFact synthetic subject-attribute corpus
Benchmarks: CounterFact MAVEN-FACT
Metrics: linear probe AUC negative log-likelihood of correct attribute probability of correct attribute after steering chi-squared test of co-occurrence
Builds on: key-value associative memory in transformers persona hypothesis of factual inconsistency CounterFact
Closest papers by meaning
- DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models (ICLR 2024 · similarity 0.88)
- On the Origins of Linear Representations in Large Language Models (ICML 2024 · similarity 0.87)
- On Linear Representations and Pretraining Data Frequency in Language Models (ICLR 2025 · similarity 0.87)
- Remarkable Robustness of LLMs: Stages of Inference? (NeurIPS 2025 · similarity 0.87)
- Truth Forest: Toward Multi-Scale Truthfulness in Large Language Models through Intervention without Tuning (AAAI 2024 · similarity 0.87)
- Liars' Bench: Evaluating Lie Detectors for Language Models (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.