Research map / Large language models / Language, safety and interpretability
Arrows of Time for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that autoregressive LLMs trained forward on natural language reach lower log-perplexity than when trained backward on time-reversed text (an "Arrow of Time"), and explains it with computational hardness and sparsity arguments on synthetic data.
Problem
Whether next-token (forward) and previous-token (backward) prediction are equally learnable by autoregressive LLMs; information theory says the joint likelihood is identical, so any consistent difference in log-perplexity would reflect an asymmetry in the data distribution.
Approach
Train matched forward and time-reversed GPT/GRU/LSTM models on the same data and measure the cross-entropy gap; for large enough models and context lengths a consistent forward advantage appears across languages. Explains it via computational hardness of inverting maps (factoring example) and sparsity of linear maps versus their inverses.
Key results
- GPT2-Medium trained on French reaches FW loss 2.788 vs BW 2.862 (+2.65%), and English 2.880 vs 2.902 (+0.76%) at the end of training.
- The forward-backward gap grows with context length in English and French GPT-Medium runs, and with model size from 5M to 405M parameters.
- On a synthetic prime-product dataset, FW perplexity on rev(pq) is 4.55 nats versus 21.56 nats for BW, reflecting the asymmetry between multiplication and factoring.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Predict the next token instead of the previous token | GPT2-Medium · CC-100 English | validation loss: 2.902 -> 2.88 | Helped |
| Predict the next token instead of the previous token | GPT2-Medium · CC-100 French | validation loss: 2.862 -> 2.788 | Helped |
| Increase GPT model size | Forward and backward GPT models · CC-100 French | BW relative loss difference: +0.15% -> +2.65% | Helped |
| Increase GPT model size | Forward and backward GPT models · CC-100 English | BW relative loss difference: -0.33% -> +0.76% | Mixed |
| Increase context window length | GPT2-Medium · CC-100 English and French | BW/FW loss percentage difference | Helped |
| Predict the next token instead of the previous token | GRU and LSTM models · CC-100 English and French | final loss: -0.07% to +0.66% | Mixed |
| Predict the next token instead of the previous token | GPT2-Medium and GPT2-XL · CC-100 multilingual | final loss: +0.61% to +3.17% | Helped |
| Predict the next token instead of the previous token | GPT2-Medium · Prime multiplication dataset | total log-perplexity: 22.2 nats -> 30.2 nats | Helped |
| Predict the next token instead of the previous token | GPT · Linear language sparse updates, 2-bit flips | loss: $0.353\pm 0.008$ -> $0.347\pm 0.009$ | Helped |
| Predict the next token instead of the previous token | GPT · Linear language sparse updates, 4-bit flips | loss: $0.371\pm 0.011$ -> $0.354\pm 0.007$ | Helped |
| Predict the next token instead of the previous token | GPT · Linear language sparse updates, 6-bit flips | loss: $0.387\pm 0.013$ -> $0.367\pm 0.014$ | Helped |
| Train the BPE tokenizer on character-reversed text (ablation) | GPT2-Medium · CC-100 Greek and French | loss | No clear effect |
Surprising findings
- Smaller backward models sometimes outperform smaller forward models on English (GRU/LSTM small sizes), though this disappears for larger models and contexts.
- The forward-backward gap is consistent across 11 languages and several architectures, yet its magnitude varies substantially by language.
Limitations
- Most experiments are single training runs; error bars were computed only for Greek with GPT2-Medium.
- Mechanistic explanation for natural language is heuristic; the synthetic examples do not establish the cause in real text.
- Experiments are limited to models up to 1.6B parameters and about one epoch of data per language.
Experimental setup
Datasets: CC-100 synthetic prime multiplication dataset synthetic linear language dataset
Benchmarks: CC-100 validation loss (English, French, German, Turkish, Finnish, Vietnamese, Greek, Indonesian, Tagalog, Arabic, Hebre
Metrics: cross-entropy / log-perplexity relative BW-FW loss difference
Baselines: GRU LSTM
Builds on: GPT-2 minGPT BPE AdamW Shannon's forward/backward prediction experiments
Closest papers by meaning
- Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale (NeurIPS 2025 · similarity 0.89)
- Correlation Dimension of Autoregressive Large Language Models (NeurIPS 2025 · similarity 0.89)
- Large language models suffer from a curse of ambiguity (NeurIPS 2026 · similarity 0.89)
- A Probabilistic Perspective on Unlearning and Alignment for Large Language Models (ICLR 2025 · similarity 0.89)
- Language Models Are Implicitly Continuous (ICLR 2025 · similarity 0.88)
- Beyond Autoregression: Fast LLMs via Self-Distillation Through Time (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.