Research map / Large language models / Retrieval and knowledge
Multiple Streams of Knowledge Retrieval: Enriching and Recalling in Transformers
Todd Nief, David Reber, Sean M. Richardson, Ari Holtzman
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the output projection O matrix to FFN grafting at the final token (ablation) | component grafting, Gemma · Relation completion headline test | top-5 accuracy: removing the O matrix and only using the FFN harms top-5 accuracy by 29% | Improved |
| add the output projection O matrix to FFN grafting at the final token (ablation) | component grafting, Llama3 · Relation completion headline test | top-5 accuracy: removing the O matrix and only using the FFN harms top-5 accuracy by 41% | Improved |
| graft task ATTN at first entity in hybrid model (ablation) | hybrid grafting of Gemma and Llama3 · Relation completion headline test | top-5 accuracy: 63% top-5 accuracy for Gemma and 34% for Llama3 | Improved |
| graft task FFNs at first entity instead of task ATTN (ablation) | hybrid grafting · Relation completion headline test | top-5 accuracy | No clear effect |
| graft both FFN up and down projections together with O matrix rather than either alone (ablation) | component grafting at final token · Relation completion headline test | top-5 accuracy | Improved |
| graft the complement of first entity and last token positions (ablation) | position grafting, all models · Relation completion headline test | top-5 accuracy: near-zero top-5 accuracy, comparable to pretrained model | Hurt |
| add movie title to last token grafting (ablation) | position grafting · Relation completion with movie title in test sentence | top-5 accuracy | Improved |
| add movie title to first entity grafting (ablation) | position grafting · Relation completion with movie title in test sentence | top-5 accuracy | Mixed |
| use finetuned instead of pretrained unembeddings (ablation) | position grafting · Relation completion headline test | top-5 accuracy | No clear effect |
| use less aggressive finetuning (lower learning rate, no weight decay, supplemental data) (ablation) | Gemma position grafting · Relation completion headline test | top-5 accuracy | Hurt |
Closest papers by meaning
- Physics of Language Models: Part 3.1, Knowledge Storage and Extraction (ICML 2024 · similarity 0.88)
- The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and More (NeurIPS 2024 · similarity 0.88)
- Too Late to Recall: Explaining the Two-Hop Problem in Multimodal Knowledge Retrieval (NeurIPS 2025 · similarity 0.87)
- Understanding Finetuning for Factual Knowledge Extraction (ICML 2024 · similarity 0.87)
- Relation Also Knows: Rethinking the Recall and Editing of Factual Associations in Auto-Regressive Transformer Language Models (AAAI 2025 · similarity 0.87)
- How new data permeates LLM knowledge and how to dilute it (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
