Research map / Large language models / Language, safety and interpretability
Language Models Are Implicitly Continuous
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends causal Transformers to continuous time (stepwise-constant token durations) and continuous embedding space, showing pretrained LLMs respond smoothly to duration changes and embedding interpolations without weight changes.
Problem
Language is modelled as discrete token sequences, yet LLMs are continuous smooth function approximators; the paper asks whether pretrained LLMs implicitly treat language as continuous in time and space.
Approach
Generalise causal attention to a Continuous Causal Transformer by replacing the sum over positions with an integral over a stepwise-constant input, so token durations and interpolated embeddings can be fed to unmodified pretrained LLMs via floating-point positions and custom attention masks.
Key results
- On a 200-sentence word-counting task, the observed normalised peak frequency is 0.7404 versus a counterfactual 0.2600 (average ratio 2.90) across six models.
- On the tutorial step-counting task, observed normalised peak frequency is 0.6706 versus counterfactual 0.2186 (average ratio 3.28).
- In the sums experiment, property P1 (shrunk-label probability exceeds original) holds for 87.82% of valid records globally.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Vary the duration of repeated tokens | Pretrained LLMs · Single-token counting | Normalised peak frequency: 0.2600 -> 0.7404 | Helped |
| Vary the duration of tutorial steps | Pretrained LLMs · Event counting | Normalised peak frequency: 0.2186 -> 0.6706 | Helped |
| Shrink the duration of a number in a two-digit sum | Pretrained LLMs · Two-digit sums | P1 frequency: 87.82% | Helped |
| Shrink the duration of a number in a two-digit sum | Pretrained LLMs · Two-digit sums | P2 frequency: 86.97% | Helped |
| Shrink the duration of a number in a two-digit sum | Pretrained LLMs · Two-digit sums | P3 frequency: 79.05% | Helped |
| Interpolate between the embeddings of apples and bananas | Llama 3 8B and other pretrained LLMs · Spatial interpolation: "Are [interpolation] red?" | Output transition between yes and no | Mixed |
| Interpolate between the embeddings of apples and bananas | Llama 3 8B and other pretrained LLMs · Spatial interpolation: "Are [interpolation] fruits?" | Fruit classification | No clear effect |
Surprising findings
- LLMs treat fractional token durations as meaningful, returning counts between integers (e.g. 'half an apple'), which humans would not do without instruction.
- Interpolated embeddings between 'apples' and 'bananas' yield coherent outputs (e.g. the intermediate fruit is predicted green for some models) even though they map to no known token.
Limitations
- Results are restricted to stepwise-constant inputs and a small set of hand-built templates.
- Many experiments rely on prompt templates with single-digit answer coaxing; tokenization mismatches excluded some records (e.g. Llama 3 treats digits as one token).
Experimental setup
Datasets: 200 word-counting sentences (synthetic template) sequential how-to tutorial dataset (200 tutorials) 100 sum word problems (synthetic)
Metrics: normalised peak frequency average ratio of observed to counterfactual peaks valid ratio P1/P2/P3 property frequencies
Builds on: Transformer Neural ODE
Closest papers by meaning
- Emergence of a High-Dimensional Abstraction Phase in Language Transformers (ICLR 2025 · similarity 0.89)
- Flexible Language Modeling in Continuous Space with Transformer-based Autoregressive Flows (NeurIPS 2025 · similarity 0.88)
- Implicit Language Models are RNNs: Balancing Parallelization and Expressivity (ICML 2025 · similarity 0.88)
- Arrows of Time for Large Language Models (ICML 2024 · similarity 0.88)
- Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale (NeurIPS 2025 · similarity 0.87)
- The Emergence of Abstract Thought in Large Language Models Beyond Any Language (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.