Research map / Large language models / Language, safety and interpretability
On Linear Representations and Pretraining Data Frequency in Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that linear relational embeddings for factual relations form when subject-object co-occurrence frequencies in pretraining data pass model-specific thresholds, and that LRE quality can be used to regress pretraining term frequencies across models.
Problem
What factors determine whether LMs form linear representations of factual relations, and can linear representation strength reveal information about frequencies in the otherwise-unknown pretraining data?
Approach
Links linear relational embedding quality across pretraining checkpoints to subject-object co-occurrence counts computed within the exact training batches (Batch Search), finding consistent frequency thresholds, and trains a random forest on LRE metrics to predict term frequencies that transfers across models.
Key results
- Co-occurrence frequency correlates with causality at r=0.82 in OLMo-7B, versus 0.66 for subject frequency and 0.59 for object frequency (OLMo-1B reported for the latter two).
- Linear representations (causality above 0.9) consistently form at about 1k co-occurrences in OLMo-7B and 2k in GPT-J, and around 4k in OLMo-1B.
- A random forest on LRE features predicts object occurrence within one order of magnitude about 70% of the time, and a regression fit on OLMo transfers to GPT-J with 0.65 object within-magnitude accuracy versus 0.31 for the mean baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use subject-object co-occurrence frequency rather than subject or object frequency alone | OLMo-7B and OLMo-1B factual-relation LREs · Factual relations | Pearson correlation with causality: 0.82 vs. 0.66 (subject frequency) and 0.59 (object frequency) | Helped |
| Use subject-object frequency rather than 5-shot accuracy as the correlate of causality | OLMo-7B factual-relation LREs · Factual relations | correlation with causality: 0.82 v.s. 0.74 | Helped |
| Use LRE features instead of LM-only features to predict object frequency | Random forest frequency regression · Heldout relations | mean absolute error (natural log space): 2.1, (4.2) | Helped |
| Use LRE features instead of LM-only features to predict subject-object frequency | Random forest frequency regression · Heldout relations | mean absolute error (natural log space): 1.9, (2.3) | Helped |
| Use LRE features instead of LogProb Features | Frequency regression trained on OLMo-7B and evaluated on GPT-J · Object occurrences | within-magnitude accuracy: 0.42±0.10 -> 0.65±0.12 | Helped |
| Use LRE features instead of LogProb Features | Frequency regression trained on GPT-J and evaluated on OLMo · Object occurrences | within-magnitude accuracy: 0.41±0.09 -> 0.49±0.12 | Helped |
| Use LRE features instead of LogProb Features | Frequency regression trained on OLMo-7B and evaluated on GPT-J · Subject-object co-occurrences | within-magnitude accuracy: 0.66±0.09 -> 0.76±0.12 | Helped |
| Use LRE features instead of LogProb Features | Frequency regression trained on GPT-J and evaluated on OLMo · Subject-object co-occurrences | within-magnitude accuracy: 0.60±0.07 -> 0.68±0.08 | Helped |
| Add hard causality as a feature | Frequency regression predicting object frequency · Heldout relations | accuracy: about 15% | Helped |
| Fit LREs using incorrect examples rather than only correct examples (ablation) | OLMo-7B LREs · Relations with enough incorrect examples | causality and faithfulness | No clear effect |
| Reach higher subject-object co-occurrence frequency during pretraining | OLMo-7B and GPT-J factual relations · Factual relations | causality: 1k and 2k times | Helped |
| Reach the co-occurrence threshold earlier rather than later in pretraining | OLMo checkpoints · Factual relations | causality | No clear effect |
Surprising findings
- Using incorrectly predicted examples to fit LREs works about as well as using only correct examples.
- Star-constellation-name has 84% 5-shot accuracy but only 44% causality in OLMo-7B, showing accuracy and linear representation can dissociate.
Limitations
- Cannot make causal claims about how exposure affects representations since counterfactual pretraining is too costly.
- Subject-object co-occurrence frequency is hard to predict; the method performs worse there than for object frequency.
- Generalization guarantees are unclear beyond the models and settings tested; only three models are studied.
Experimental setup
Datasets: Relations dataset Dolma The Pile
Benchmarks: Relations dataset
Metrics: causality faithfulness Pearson correlation within-magnitude accuracy 5-shot accuracy mean absolute error
Baselines: LogProb features Mean frequency baseline 5-shot accuracy
Builds on: Linear Relational Embeddings What's in My Big Data (WIMBD) OLMo Dolma
Closest papers by meaning
- Scalable Influence and Fact Tracing for Large Language Model Pretraining (ICLR 2025 · similarity 0.89)
- On the Origins of Linear Representations in Large Language Models (ICML 2024 · similarity 0.88)
- Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale (NeurIPS 2025 · similarity 0.88)
- Analyzing Similarity Metrics for Data Selection for Language Model Pretraining (NeurIPS 2025 · similarity 0.87)
- Emergence of Linear Truth Encodings in Language Models (NeurIPS 2025 · similarity 0.87)
- Less is More: Local Intrinsic Dimensions of Contextual Language Models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.