Research map / Large language models / Language, safety and interpretability
Language Models Represent Space and Time
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Probes Llama-2 (7B-70B) and Pythia activations on six new space and time datasets, finding linear, prompt-robust, entity-unified representations of latitude/longitude and years, plus individual space and time neurons.
Problem
Whether LLMs learn coherent, grounded representations of real-world spatial and temporal coordinates or only superficial statistics.
Approach
Extract actual world-map and timeline coordinates from LLM activations by training linear ridge probes on entity-name activations across layers, at multiple spatiotemporal scales, and then test linearity, prompt robustness, generalization, and neuron-level use with ablations and interventions.
Key results
- Linear probes on Llama-2-70b reach out-of-sample R^2 of 0.911 on world places and 0.885 on entertainment dates at 60% depth, with NYC lowest at 0.359.
- A one-layer MLP probe improves R^2 minimally over linear probes (e.g., 0.911 vs 0.926 on world for 70b), supporting linear representation.
- Individual space and time neurons (e.g., L19.3610 with 0.77 correlation to release date) exist; ablating or pinning them changes loss or next-token predictions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the linear probe with a one-layer MLP with 256 neurons (ablation) | Llama-2-7B probes · World, USA, NYC, Historical, Entertainment, and Headlines | R²: World: 0.881 -> 0.897; USA: 0.799 -> 0.819; NYC: 0.219 -> 0.204; Historical: 0.785 -> 0.775; Entertainment: 0.788 -> 0.7 | Mixed |
| Replace the linear probe with a one-layer MLP with 256 neurons (ablation) | Llama-2-13B probes · World, USA, NYC, Historical, Entertainment, and Headlines | R²: World: 0.896 -> 0.916; USA: 0.825 -> 0.824; NYC: 0.237 -> 0.230; Historical: 0.804 -> 0.818; Entertainment: 0.806 -> 0.8 | Mixed |
| Replace the linear probe with a one-layer MLP with 256 neurons (ablation) | Llama-2-70B probes · World, USA, NYC, Historical, Entertainment, and Headlines | R²: World: 0.911 -> 0.926; USA: 0.864 -> 0.869; NYC: 0.359 -> 0.312; Historical: 0.835 -> 0.839; Entertainment: 0.885 -> 0.8 | Mixed |
| Zero ablate space neuron L20.7573 (ablation) | Llama-2-7B · Wikipedia place and entertainment contexts | next-token loss: 2.107 | Hurt |
| Zero ablate space neuron L20.7423 (ablation) | Llama-2-7B · Wikipedia place and entertainment contexts | next-token loss: 1.971 | Hurt |
| Zero ablate time neuron L18.9387 (ablation) | Llama-2-7B · Wikipedia place and entertainment contexts | next-token loss: 2.254 | Hurt |
| Zero ablate time neuron L19.3610 (ablation) | Llama-2-7B · Wikipedia place and entertainment contexts | next-token loss: 1.082 | Hurt |
| Probe the period token appended to each headline instead of the headline's last token | Llama-2-70B headline probes · Headlines | R² | Helped |
Surprising findings
- Random distracting tokens before the entity substantially degrade probe performance, while explicit recall prompts make little difference.
- Block holdout generalization is clearly above random but much worse than nominal, suggesting probes memorize absolute coordinate mapping.
Limitations
- English Wikipedia skews data toward the Anglosphere and entity types are non-uniform.
- Probing shows decodability, not that the model uses the representations; the causal evidence is limited to a few neurons.
- Spatiotemporal representations are not a dynamic causal world model.
Experimental setup
Datasets: DBpedia Wikipedia pageviews NYC OpenData points of interest historical figures dataset New York Times headlines
Benchmarks: World places USA places NYC places Historical figures Art and entertainment Headlines
Metrics: out-of-sample R^2 Spearman rank correlation proximity error
Baselines: MLP probe Pythia
Builds on: linear representation hypothesis Othello-GPT probing factual recall localization
Closest papers by meaning
- LTD-Bench: Evaluating Large Language Models by Letting Them Draw (NeurIPS 2025 · similarity 0.88)
- Language Models Are Implicitly Continuous (ICLR 2025 · similarity 0.87)
- Less is More: Local Intrinsic Dimensions of Contextual Language Models (NeurIPS 2025 · similarity 0.87)
- Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces (CVPR 2025 · similarity 0.87)
- SpaCE-10: A Comprehensive Benchmark for Multimodal Large Language Models in Compositional Spatial Intelligence (ICLR 2026 · similarity 0.86)
- Episodic Memories Generation and Evaluation Benchmark for Large Language Models (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.