Research map / Large language models / Retrieval and knowledge
Relation Also Knows: Rethinking the Recall and Editing of Factual Associations in Auto-Regressive Transformer Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a relation-focused interpretation of factual recall in autoregressive LMs and RETS, a ROME-style single-fact editing method that edits the middle-late MLP at the last relation token with subject constraints to reduce over-generalizing.
Problem
Locate-then-edit knowledge editing methods (ROME, PMET) edit at the last-subject token and ignore relation information, causing over-generalizing where unrelated relations of the edited subject are unexpectedly changed.
Approach
Interpret factual recall via relation tokens: relational knowledge is aggregated at the last-relation token through early-to-middle-late MLP layers, so editing the middle-late MLP at that token (with least-squares constraints on neighboring subjects) avoids over-generalizing while keeping subject specificity.
Key results
- RETS achieves R-Specificity of 78.5 on GPT2-XL versus 41.1 for ROME and 47.6 for PMET, over 30 points higher.
- On GPT-J, RETS reaches R-Specificity 83.3 compared with 48.5 for ROME and 55.6 for PMET while keeping Efficacy at 100.0.
- RETS trades off Generalization (71.5 vs 96.4 for ROME on GPT2-XL) and S-Specificity (68.6 vs 76.0) for better R-Specificity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use relation-focused RETS editing instead of subject-focused PMET | GPT2-XL · COUNTERFACT | R-Specificity: 47.6 -> 78.5 | Helped |
| Use relation-focused RETS editing instead of subject-focused PMET | GPT-J · COUNTERFACT | R-Specificity: 55.6 -> 83.3 | Helped |
| Use relation-focused RETS editing instead of ROME | Llama-2 · COUNTERFACT | R-Specificity: 53.6 -> 83.1 | Helped |
| Remove subject constraints from RETS (ablation) | RETS on GPT2-XL · COUNTERFACT | S-Specificity: 68.6 -> 35.1 | Hurt |
| Add subject constraints to relation-focused RETS editing (ablation) | RETS on GPT-J · COUNTERFACT | S-Specificity: 23.7 -> 65.5 | Helped |
| Remove subject constraints from RETS (ablation) | RETS on GPT2-XL · COUNTERFACT | Fluency: 577.4 -> 626.1 | Helped |
| Remove subject constraints from RETS (ablation) | RETS on GPT-J · COUNTERFACT | Fluency: 542.4 -> 618.1 | Helped |
| Remove subject constraints from RETS (ablation) | RETS on GPT2-XL · COUNTERFACT | R-Specificity: 78.5 -> 86.9 | Helped |
| Remove subject constraints from RETS (ablation) | RETS on GPT-J · COUNTERFACT | R-Specificity: 83.3 -> 90.7 | Helped |
| Block MLP updates rather than MHSA updates at the last token (ablation) | GPT2-XL relation representation · Relation attributes across layers | Attributes rate | Hurt |
| Edit at the last-relation token in middle-late layers rather than the last-subject token | RETS · COUNTERFACT | R-Specificity | Helped |
| Use RETS instead of ROME | GPT2-XL · zsRE | Generalization: 87.8 -> 77.9 | Hurt |
Surprising findings
- Blocking MLP sublayers at the last-relation token causes a much larger drop in relational attributes rate than blocking MHSA, and the first-relation/prefix tokens contribute little despite the relation being spread over several tokens.
- Ablating subject constraints in RETS lowers S-Specificity sharply (68.6 to 35.1 on GPT2-XL) while raising R-Specificity, suggesting a trade-off between the two.
Limitations
- Subject constraints are simple and degrade Fluency and Consistency of generated text.
- Only single-fact editing is studied; scaling to sequential or batched edits is not addressed.
- Analysis and editing focus only on MLP sublayers, leaving other components unexplored.
Experimental setup
Datasets: COUNTERFACT zsRE Wikidata attribute collections
Benchmarks: COUNTERFACT zsRE
Metrics: Efficacy Generalization S-Specificity R-Specificity Fluency Consistency Specificity
Baselines: FT-L MEND ROME PMET MEMIT
Builds on: ROME MEMIT PMET causal tracing vocabulary lens
Closest papers by meaning
- Representation Shattering in Transformers: A Synthetic Study with Knowledge Editing (ICML 2025 · similarity 0.90)
- Linearity of Relation Decoding in Transformer Language Models (ICLR 2024 · similarity 0.89)
- Knowledge in Superposition: Unveiling the Failures of Lifelong Knowledge Editing for Large Language Models (AAAI 2025 · similarity 0.88)
- Uncovering Overfitting in Large Language Model Editing (ICLR 2025 · similarity 0.88)
- WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models (NeurIPS 2024 · similarity 0.88)
- Unveiling the Pitfalls of Knowledge Editing for Large Language Models (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.