AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

Transformer as a hippocampal memory consolidation model based on NMDAR-inspired nonlinearity

Dong-Kyum Kim, Jea Kwon, Meeyoung Cha, C. Justin Lee

NeurIPS 2023 · Architectures and efficiency › RNNs and normalization · approach: Mechanistic circuit interpretability · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
set NMDA alpha from 0 to larger values in the FFN activation (ablation)TransformerXL with NMDA activation · 2D grid navigation, training mapsreference memory errorImproved
remove nonlinearity (alpha = 0) in the FFN activation (ablation)TransformerXL with NMDA activation · 2D grid navigationworking memory errorNo clear effect
evaluate reference memory on novel maps for all alphaTransformerXL with NMDA activation · 2D grid navigation, novel mapsreference memory error: chance level of 0.9No clear effect
increase number of training maps N from 32 to 48 to 64TransformerXL with NMDA activation · 2D grid navigation, training mapsreference memory errorHurt
use NMDA alpha=10 instead of GELU, ReLU, LeakyReLU, sigmoid, tanhTransformerXL FFN · 2D grid navigation, training mapsreference memory errorImproved
replace recurrent positional embedding with non-recurrent action embedding (ablation)TransformerXL with NMDA activation · 2D grid navigationworking and reference memory errorHurt
increase alpha under non-recurrent positional embedding (ablation)TransformerXL with NMDA activation · 2D grid navigationreference memory errorImproved
increase alpha in NMDA activationFFN layer neurons · 2D grid navigation, place cell scoreplace cell scoreImproved
use LeakyReLU with tuned negative slope instead of NMDA alpha=10TransformerXL FFN · 2D grid navigationaverage place cell scoreHurt
raise learning rate from 0.0001 up to 0.001 at low alpha (ablation)TransformerXL with NMDA activation · 2D grid navigation, N=64reference memory errorNo clear effect
replace GELU with NMDA alpha=10 in GPT2 124MGPT2 (nanoGPT) · OpenWebTextvalidation lossImproved
replace GELU with NMDA alpha=10 in ViTViT · CIFAR100top-1 accuracy: 69.92 -> 70.27No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy