AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Birth of a Transformer: A Memory Viewpoint

Alberto Bietti, Vivien Cabannes, Diane Bouchacourt, Herve Jegou, Leon Bottou

NeurIPS 2023 · Architectures and efficiency › Attention and transformers · approach: Transformer expressivity and learnability · 9 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use two transformer layers instead of oneVanilla transformer · Synthetic bigram data, fixed triggersaccuracy: around 55% -> over 99%Improved
Use two transformer layers instead of oneVanilla transformer · Synthetic bigram data, random triggersaccuracy: around 55% -> 95%Improved
Train with uniform output tokens rather than outputs sampled from global bigram distributions (ablation)Two-layer transformer · Synthetic bigram task, out-of-distribution output tokensin-context accuracyImproved
Use fewer trigger tokens (ablation)Two-layer transformer · Synthetic bigram taskin-context learning speedHurt
Use only rare fixed triggers (ablation)Two-layer transformer · Synthetic bigram taskin-context learning speedHurt
Use random triggers rather than fixed triggers (ablation)Two-layer transformer · Synthetic bigram taskin-context learning speedHurt
Freeze the output matrix W_O^2 while training key-query matrices (ablation)Simplified two-layer transformer induction head · Synthetic trigger-output predictionmemory recallHurt
Train W_O^2 before the key-query matrices (ablation)Simplified two-layer transformer induction head · Synthetic trigger-output predictionin-context accuracyImproved
Train W_K^2 before W_K^1 (ablation)Simplified two-layer transformer induction head · Synthetic trigger-output predictionmemory recallImproved
Train W_K^1 on early context positions rather than late positions (ablation)Simplified two-layer transformer induction head · Synthetic trigger-output predictionmemory recallImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy