AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Block-Recurrent Transformers

DeLesley Hutchins, Imanol Schlag, Yuhuai Wu, Ethan Dyer, Behnam Neyshabur

NeurIPS 2022 · Architectures and efficiency › Attention and transformers · approach: Sparse efficient transformers for long context · 28 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add one block-recurrent layer instead of a 13th non-recurrent layerSlide:13L · PG19 test (tokens)bits per token: 3.58 -> 3.53Improved
add block recurrence instead of a 13th layer on arXivSlide:13L · arXivbits per token: 1.42 -> 1.24Improved
use fixed gate instead of LSTM gate (skip config vs lstm dual) (ablation)Block-Recurrent Transformer · arXivbits per token: 1.26 -> 1.24Improved
increase recurrent states from 512 to 1024 (ablation)Rec:fixed:skip · PG19 test (tokens)bits per token: 3.53 -> 3.51Improved
increase recurrent states from 1024 to 2048 (ablation)Rec:fixed:skip · PG19 test (tokens)bits per token: 3.51 -> 3.55Hurt
add a second recurrent layer (adjacent layers 9 and 10) (ablation)Block-Recurrent Transformer · PG19perplexityNo clear effect
add block feedback (all layers cross-attend to recurrent states) (ablation)Rec:fixed:skip · PG19 test (tokens)bits per token: 3.53 -> 3.49Improved
shrink window size from 512 to 128 (ablation)Rec:fixed:skip · PG19 test (tokens)bits per token: 3.53 -> 3.58Hurt
use recurrence over larger window XL:2048 vs recurrenceXL:2048 · GitHub validationbits per token: 1.01 -> 0.976Improved
add recurrence at 1.3B scale26-layer Transformer-XL · PG19 test (tokens)bits per token: 3.31 -> 3.22Improved
train sliding window over 4096 segment (differentiable) instead of Transformer-XL 512XL:512 · PG19 test (tokens)bits per token: 3.62 -> 3.60Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy