Research map / Graph learning / Knowledge and temporal graphs

Representation Shattering in Transformers: A Synthetic Study with Knowledge Editing

ICML 2025 · Graph learning › Knowledge and temporal graphs · approach: Temporal knowledge graph reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a synthetic knowledge-graph task with a small Transformer to show that knowledge editing distorts entity representations ("representation shattering"), which correlates with accuracy loss on recall, logical and compositional inference, and validates the finding briefly on Llama 3 8B Instruct and Mamba.

Problem

Knowledge editing (KE) methods can degrade a model's broader factual recall and reasoning, but the mechanistic reason why such harms occur is not understood.

Approach

Train a small Transformer from scratch on random walks over a structured knowledge graph made of cyclic relation orders, so that editing one fact has trickling effects on related entities; measure how KE distorts the pairwise geometry of entity representations (representation shattering) and show this geometry distortion tracks accuracy loss and counterfactual edit distance.

Key results

• Corrective edits drop direct recall accuracy by about 22 points on average with ROME in the synthetic setting, and counterfactual edits at distance 4 drop direct recall by about 78 points.
• Representation shattering metric R(D*) correlates strongly with accuracy drop (r^2 = 0.905) across corrective ROME edits.
• On Llama 3 8B Instruct with MEMIT batch counterfactual month edits, MMLU-Redux reasoning accuracy declines gradually as edit distance increases from 1 to 3, and latent representations shatter more.

Measured effects reported in the paper

Surprising findings

• Corrective edits, which target facts the model recalls incorrectly, reduce accuracy on all facts rather than improving them.

Limitations

• Main results rely on a 2-layer toy Transformer and a synthetic cyclic knowledge graph, which may not transfer to larger models.
• Causal link between shattering and performance loss is supported by correlation and manipulation of edit distance, without formal proof.
• LLM validation is limited to month-order counterfactual edits and a single benchmark for Llama 3 and Mamba.

Experimental setup

Datasets: synthetic cyclic knowledge graph (2048 entities, 3 cyclic orders, 24 relations) MMLU-Redux

Benchmarks: MMLU-Redux synthetic direct recall synthetic logical inference synthetic compositional inference

Metrics: accuracy change in accuracy representation shattering metric R(D*) r^2

Baselines: ROME MEMIT AlphaEdit PMET

Builds on: ROME MEMIT nanoGPT

Closest papers by meaning

• Relation Also Knows: Rethinking the Recall and Editing of Factual Associations in Auto-Regressive Transformer Language Models (AAAI 2025 · similarity 0.90)
• Exposing the Illusion of Erasure in Knowledge Editing for LLMs (NeurIPS 2026 · similarity 0.89)
• Knowledge Circuits in Pretrained Transformers (NeurIPS 2024 · similarity 0.88)
• Unveiling the Pitfalls of Knowledge Editing for Large Language Models (ICLR 2024 · similarity 0.86)
• Generalization or Hallucination? Understanding Out-of-Context Reasoning in Transformers (NeurIPS 2025 · similarity 0.86)
• Knowledge in Superposition: Unveiling the Failures of Lifelong Knowledge Editing for Large Language Models (AAAI 2025 · similarity 0.86)