AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Recurrent Memory Transformer

Aydar Bulatov, Yuri Kuratov, Mikhail Burtsev

NeurIPS 2022 · Architectures and efficiency › Attention and transformers · approach: Sparse efficient transformers for long context · 25 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
increase BPTT unroll from 0 to 3 (ablation)RMT mem 10 · WikiText-103 seg 150perplexity: 26.85 -> 25.04Improved
increase BPTT unroll from 0 to 1 for mem 25 (ablation)RMT mem 25 · WikiText-103 seg 150perplexity: 29.73 -> 24.91Improved
increase BPTT unroll from 0 to 4 (ablation)RMT mem 10 · WikiText-103 seg 50perplexity: 30.69 -> 26.25Improved
increase BPTT unroll from 0 to 2 (ablation)RMT mem 20 · enwik8bpc: 1.301 -> 1.222Improved
add RMT memory to Tr-XL cacheTr-XL · WikiText-103 seg 150perplexity: 24.12 -> 23.99Improved
replace Tr-XL cache with RMT memory (mem 10 vs 75)Tr-XL · WikiText-103 seg 150perplexity: 24.68 (Tr-XL 75) vs 25.04 (RMT 10)No clear effect
replace Memory Transformer with RMT recurrenceMemTr · WikiText-103 seg 150perplexity: 29.63 -> 25.04Improved
use RMT instead of Tr-XL on multi-segment quadratic equationsTr-XL · Quadratic equationsaccuracy: 0.93 -> 0.99Improved
increase number of segments in Copy/ReverseTransformer-XL vs RMT · Copy/ReverseaccuracyImproved
increase memory size from 5 to 50 (ablation)RMT · WikiText-103 seg 50perplexityNo clear effect
add RMT memory of 10 tokens to pretrained models with up to 4 segmentsRoBERTa-base · Hyperpartisan newsF1: 94.87 -> 98.11Improved
add RMT memory with more segmentsBERT / DeBERTa · Hyperpartisan newsF1: deberta 96.78 (2 seg) -> 94.80 (3-4 seg)Mixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy