Research map / Large language models / Language, safety and interpretability
Lines of Thought in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Treats LLM hidden-state trajectories of the last token as ensemble "lines of thought," finds they lie on a ~256-dimensional curved manifold, and models them with a Langevin/Fokker-Planck stochastic equation with few fitted parameters.
Problem
Understanding the internal transport of token representations through transformer layers at an ensemble level, and whether the high-dimensional trajectories can be described by a simple low-parameter model.
Approach
Treats the last-token hidden states across layers as trajectories; uses ensemble SVD bases per layer to show low-dimensional clustering, then models each layer-to-layer step as a linear rotation-and-stretch plus Gaussian noise whose variance scales as exp(lambda(t+tau)), extended to a continuous Langevin SDE.
Key results
- Trajectories of pilot tokens are well described by about 256 principal components (25% of 1024 dims) for next-token prediction in GPT-2, with KL around 10% of the unrelated baseline.
- Residuals of the linear extrapolation are Gaussian, zero-mean, isotropic, with variance scaling as exp(lambda(t+tau)); fitted alpha ~0.64, lambda ~0.18 for GPT-2.
- A linear SVM separates true from extrapolated positions with only 46-61% accuracy, near chance; the same pattern holds for Llama 2 7B and Mistral 7B, with a last-layer anomaly in Mistral and Llama 3.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Approximate later token positions using ensemble-derived rotation and stretch, with stochastic residuals | GPT-2 medium · Walden, true versus extrapolated positions across layers | linear-classifier accuracy: 48–61; random guessing: 50% | No clear effect |
| Keep only the first 256 principal components at each layer (ablation) | GPT-2 medium trajectory representations · Next-token output distribution | KL divergence relative to uncorrelated baseline: about 10% | Helped |
| Reinitialize all model weights (ablation) | GPT-2 medium · Walden trajectory ensembles | fit to the proposed stochastic trajectory model | Hurt |
| Reinitialize layers 13–24 (ablation) | GPT-2 medium · Walden trajectories through layers 13–24 | trajectory transport | Hurt |
| Use trained rather than reinitialized weights | GPT-2 medium · Walden trajectory ensembles | fit to the proposed stochastic trajectory model | Helped |
| Use Mistral 7B v0.1 | Mistral 7B v0.1 · Walden trajectories, final layer | linear-extrapolation error | Hurt |
| Use Llama 3.2 1B or 3B | Llama 3.2 · Walden trajectories, final layer | residual mean and variance | Mixed |
Surprising findings
- Untrained or partially reinitialized models yield straight, quasi-parallel trajectories or trajectories that stop, so the structure emerges only from training.
- The last layer is anomalous in Mistral 7B and in all Llama 3.2 models tested.
Limitations
- Limited to open-weight models because hidden states must be extracted.
- Linear approximation with isotropic Gaussian noise ignores variability across times and dimensions; the stochastic model is only approximate.
- Last-layer normalization complicates linking the latent trajectories to the output vocabulary distribution.
Experimental setup
Datasets: Walden (Henry David Thoreau, Project Gutenberg)
Metrics: KL divergence between truncated and full output distribution mean over standard deviation of residuals excess kurtosis variance scaling of residuals linear SVM separability accuracy
Baselines: untrained (reinitialized) GPT-2 gibberish inputs ablated GPT-2 with reinitialized layers 13-24
Builds on: neural ODE transformer residual stream
Closest papers by meaning
- Correlation Dimension of Autoregressive Large Language Models (NeurIPS 2025 · similarity 0.88)
- Arrows of Time for Large Language Models (ICML 2024 · similarity 0.87)
- Large language models suffer from a curse of ambiguity (NeurIPS 2026 · similarity 0.87)
- Less is More: Local Intrinsic Dimensions of Contextual Language Models (NeurIPS 2025 · similarity 0.86)
- Language Models Are Implicitly Continuous (ICLR 2025 · similarity 0.86)
- On the Origins of Linear Representations in Large Language Models (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.