AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Memorizing Transformers

Yuhuai Wu, Markus Norman Rabe, DeLesley Hutchins, Christian Szegedy

ICLR 2022 · Architectures and efficiency › Attention and transformers · approach: Efficient language transformers · 40 citations (OpenAlex)

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Add 65K external memoryTransformer-XL · arXiv Mathperplexity: 2.42 -> 2.26Improved
Add 65K external memoryTransformer-XL · PG-19perplexity: 11.88 -> 11.37Improved
Add 65K external memoryTransformer-XL · C4(4K+)perplexity: 14.03 -> 13.64Improved
Add 65K external memoryTransformer-XL · GitHubperplexity: 2.10 -> 1.80Improved
Add 65K external memoryTransformer-XL · Isabelleperplexity: 2.16 -> 1.99Improved
Increase memory size from 8192 to 65KMemorizing Transformer, context 512 with XL cache 512 · arXiv Mathperplexity: 2.37 -> 2.31Improved
Increase memory size from 8192 to 65KMemorizing Transformer, context 512 with XL cache 512 · PG-19perplexity: 11.93 -> 11.62Improved
Increase memory size from 8192 to 65KMemorizing Transformer, context 512 with XL cache 512 · GitHubperplexity: 2.03 -> 1.87Improved
Increase memory size from 8192 to 65KMemorizing Transformer, context 512 with XL cache 512 · C4(4K+)perplexity: 14.04 -> 14.04No clear effect
Use two kNN retrieval layers rather than one (ablation)Memorizing Transformer · Language modeling tasksperplexityNo clear effect
Place the memory layer at layer 3 rather than layer 9 (ablation)Memorizing Transformer · arXiv Mathperplexity: 2.37 -> 2.40Hurt
Place the memory layer at layer 6 rather than layer 9 (ablation)Memorizing Transformer · arXiv Mathperplexity: 2.37 -> 2.36Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy