Research map / Large language models / Retrieval and knowledge

The Geometry of Forgetting: Temporal Knowledge Drift as an Independent Axis in LLM Representations

NeurIPS 2026 · Large language models › Retrieval and knowledge · approach: Knowledge editing in LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that temporal knowledge drift (whether a stored fact has changed since training) is linearly decodable in LLM residual streams as a direction geometrically orthogonal to correctness and uncertainty, so output-level and correctness/uncertainty detectors fail while a drift-supervised probe reaches AUROC 0.83-0.95.

Problem

Instruction-tuned LLMs give outdated answers with the same confidence as correct ones, and existing hallucination/uncertainty detectors cannot flag temporally stale facts. The paper asks whether temporal drift is represented internally in a way that correctness or uncertainty signals miss.

Approach

Temporal drift is encoded as a direction in the residual stream orthogonal to correctness and uncertainty; a probe trained directly on drift labels detects it, and a cross-cutoff experiment on byte-identical inputs shows the probe reads model-specific knowledge state rather than input properties.

Key results

• Drift probe reaches controlled AUROC 0.83-0.95 across six models (mean 0.89), while token entropy, semantic entropy, CCS, SAPLMA and SEP stay at 0.49-0.57 mean.
• Weight cosines between drift and correctness/uncertainty probes are at most 0.14, and null-space projection changes drift AUROC by at most 0.008.
• Cross-cutoff experiment yields P(A>B) = 0.975-0.998 across seven model pairs on byte-identical inputs.

Surprising findings

• Semantic entropy rates stale recalls as reliable because the model consistently retrieves the same outdated answer across samples.
• The MLP retrieval circuit gives near-identical trajectories for stale recall and confabulation (r > 0.81), so output confidence cannot separate them.

Limitations

• Covers only four Wikidata relations with discrete factual transitions.
• Only six instruction-tuned models of 2-9B parameters; frontier-scale behavior untested.
• The probe is supervised and requires drift-labeled examples.

Experimental setup

Datasets: Wikidata holder timelines (frozen 2026-04-27 snapshot)

Benchmarks: temporal drift dataset (controlled post-cutoff split)

Metrics: AUROC Pearson r Mann-Whitney U P(A>B) logit change trajectory correlation

Baselines: Token entropy Token probability Sequence log-prob Sequence entropy Top-k margin Semantic Entropy SEP SAPLMA CCS MLP probe (drift)

Builds on: SAPLMA CCS Semantic Entropy SEP INLP

Closest papers by meaning

• Geometry Conflict: Explaining and Controlling Forgetting in LLM Continual Post-Training (NeurIPS 2026 · similarity 0.88)
• Spurious Forgetting in Continual Learning of Language Models (ICLR 2025 · similarity 0.88)
• Knowledge Entropy Decay during Language Model Pretraining Hinders New Knowledge Acquisition (ICLR 2025 · similarity 0.88)
• Tracing the Representation Geometry of Language Models from Pretraining to Post-training (NeurIPS 2025 · similarity 0.88)
• How Do Large Language Models Acquire Factual Knowledge During Pretraining? (NeurIPS 2024 · similarity 0.86)
• Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts (ICLR 2024 · similarity 0.86)