AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Random-Access Infinite Context Length for Transformers

Amirkeivan Mohtashami, Martin Jaggi

NeurIPS 2023 · Architectures and efficiency › Attention and transformers · approach: Attention head and block redesign

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add landmark retrieval (local 250, k=2) in place of plain 360-token attention (ablation)GPT-2-like transformer · PG19, eval 512perplexity: 16.76 -> 16.23Improved
add landmark retrieval (local 250, k=2) in place of plain 360-token attention (ablation)GPT-2-like transformer · arXiv, eval 512perplexity: 4.31 -> 4.01Improved
restrict retrieved blocks to be the same across tokens (per head only) (ablation)Landmark attention · PG19, eval 4096, k=4perplexity: 14.72 -> 14.95Hurt
restrict retrieved blocks to be the same across heads (per token only) (ablation)Landmark attention · PG19, eval 4096, k=4perplexity: 14.72 -> 14.89Hurt
increase retrieved blocks k from 2 to 4 at full flexibility (ablation)Landmark attention · PG19, eval 2048perplexity: 15.14 -> 14.92Improved
increase eval length from 2048 to 4096 at k=4 (ablation)Landmark attention · PG19perplexity: 14.92 -> 14.72Improved
increase k from 2 to 4 with per-head-only retrieval (ablation)Landmark attention · PG19, eval 2048perplexity: 15.48 -> 15.10Improved
increase memory blocks from 20 to 40 at k=4, local 250 (ablation)Landmark attention · PG19, eval 2048perplexity: 15.02 -> 14.92Improved
increase local context from 250 to 350 at k=2, 40 blocks (ablation)Landmark attention · PG19, eval 2048perplexity: 15.14 -> 15.07Improved
increase memory blocks from 40 to 80 at k=4, eval 4096 (ablation)Landmark attention · PG19, eval 4096perplexity: 14.79 -> 14.72Improved
train with context miss token (CMT), no cutoff (ablation)Landmark attention · PG19, eval 2048perplexity: 16.28 -> 16.38Hurt
drop retrievals via CMT cutoff 0.5 (ablation)Landmark attention with CMT · PG19, eval 2048perplexity: 16.38 -> 16.43Mixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy