AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

Sparse Attentive Backtracking: Temporal Credit Assignment Through Reminding

Nan Rosemary Ke, Anirudh Goyal ALIAS PARTH GOYAL, Olexa Bilaniuk, Jonathan Binas, Michael Mozer, Chris Pal, Yoshua Bengio

NeurIPS 2018 · Architectures and efficiency › RNNs and normalization · approach: LSTM design and memory · 34 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use sparse attentive backtracking instead of truncated BPTTLSTM on copying sequences · Copying T=300accuracy (last 10 characters): 24.4 -> 99.9Improved
Use sparse attentive backtracking instead of full BPTTLSTM on CIFAR10 pixel-by-pixel classification · CIFAR10accuracy: 58.3 -> 64.5Improved
Use sparse attentive backtracking instead of full self-attentionLSTM on permuted MNIST · Permuted MNISTaccuracy: 97.9 -> 94.2Hurt
Train on T=100 and transfer to longer sequences with SAB rather than full-BPTT LSTMLSTM on copying memory · Copying transfer T=200accuracy (last 10 digits): 34% -> 95%Improved
Train on T=100 and transfer to longer sequences with SAB rather than full-BPTT LSTMLSTM on copying memory · Copying transfer T=5000accuracy (last 10 digits): 12% -> 41%Improved
Add mental updates that propagate gradients locally around retrieved memories (ablation)SAB on copying T=100, k_top=1 · Copying T=100accuracy: 49.0 -> 57.9Improved
Add mental updates that propagate gradients locally around retrieved memories (ablation)SAB on copying T=100, k_top=5 · Copying T=100accuracy: 98.3 -> 100.0Improved
Attend to all past events instead of a sparse subset (ablation)SAB on copying T=100 · Copying T=100accuracy: 100.0 -> 40.5Hurt
Use SAB rather than LSTM with full BPTTCharacter-level language modeling · Text8 testBPC: 1.42 -> 1.47; 1.42 -> 1.44Mixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy