Research map / Architectures and efficiency / RNNs and normalization
Transformer as a hippocampal memory consolidation model based on NMDAR-inspired nonlinearity
Dong-Kyum Kim, Jea Kwon, Meeyoung Cha, C. Justin Lee
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| set NMDA alpha from 0 to larger values in the FFN activation (ablation) | TransformerXL with NMDA activation · 2D grid navigation, training maps | reference memory error | Improved |
| remove nonlinearity (alpha = 0) in the FFN activation (ablation) | TransformerXL with NMDA activation · 2D grid navigation | working memory error | No clear effect |
| evaluate reference memory on novel maps for all alpha | TransformerXL with NMDA activation · 2D grid navigation, novel maps | reference memory error: chance level of 0.9 | No clear effect |
| increase number of training maps N from 32 to 48 to 64 | TransformerXL with NMDA activation · 2D grid navigation, training maps | reference memory error | Hurt |
| use NMDA alpha=10 instead of GELU, ReLU, LeakyReLU, sigmoid, tanh | TransformerXL FFN · 2D grid navigation, training maps | reference memory error | Improved |
| replace recurrent positional embedding with non-recurrent action embedding (ablation) | TransformerXL with NMDA activation · 2D grid navigation | working and reference memory error | Hurt |
| increase alpha under non-recurrent positional embedding (ablation) | TransformerXL with NMDA activation · 2D grid navigation | reference memory error | Improved |
| increase alpha in NMDA activation | FFN layer neurons · 2D grid navigation, place cell score | place cell score | Improved |
| use LeakyReLU with tuned negative slope instead of NMDA alpha=10 | TransformerXL FFN · 2D grid navigation | average place cell score | Hurt |
| raise learning rate from 0.0001 up to 0.001 at low alpha (ablation) | TransformerXL with NMDA activation · 2D grid navigation, N=64 | reference memory error | No clear effect |
| replace GELU with NMDA alpha=10 in GPT2 124M | GPT2 (nanoGPT) · OpenWebText | validation loss | Improved |
| replace GELU with NMDA alpha=10 in ViT | ViT · CIFAR100 | top-1 accuracy: 69.92 -> 70.27 | No clear effect |
Closest papers by meaning
- Neuroformer: Multimodal and Multitask Generative Pretraining for Brain Data (ICLR 2024 · similarity 0.85)
- The Expressive Leaky Memory Neuron: an Efficient and Expressive Phenomenological Neuron Model Can Solve Long-Horizon Tasks. (ICLR 2024 · similarity 0.85)
- Learning the Plasticity: Plasticity-Driven Learning Framework in Spiking Neural Networks (NeurIPS 2025 · similarity 0.84)
- SPICED: A Synaptic Homeostasis-Inspired Framework for Unsupervised Continual EEG Decoding (NeurIPS 2025 · similarity 0.84)
- Exponential Dynamic Energy Network for High Capacity Sequence Memory (NeurIPS 2025 · similarity 0.84)
- Shaping the distribution of neural responses with interneurons in a recurrent circuit model (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
