Research map / Architectures and efficiency / Attention and transformers
MemoryLLM: Plug-n-Play Interpretable Feed-Forward Memory for Transformers
AJAY KUMAR JAISWAL, Lauren Hannah, Han-Byul Kim, Duc N.M Hoang, Arnav Kundu, Mehrdad Farajtabar, Minsik Cho
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train FFNs on context-free token embeddings (MemoryLLM) instead of dense FFNs, compared at equal total params | dense base 750M · C4 | perplexity: 19.730 -> 22.079 | Hurt |
| MemoryLLM vs dense model at matched active parameters | dense base 250M · C4 | perplexity: 23.190 -> 22.079 | Improved |
| move beta*h^2 FFN params from memory to FFN compute (Flex-MemoryLLM-3h2) | MemoryLLM 1B · LAMBADA | accuracy: 36.13 -> 44.19 | Improved |
| move beta*h^2 FFN params from memory to FFN compute (Flex-MemoryLLM-3h2) | MemoryLLM 1B · HellaSwag | accuracy: 37.24 -> 41.64 | Improved |
| reduce FFN contribution alpha from 1.0 to 0.5 (ablation) | MemoryLLM-1B · LAMBADA | accuracy: 0.3613 -> 0.1086 | Hurt |
| reduce FFN contribution alpha from 1.0 to 0.5 (ablation) | MemoryLLM-1B · Wikitext-2 | perplexity: 24.348 -> 53.7072 | Hurt |
| reduce FFN contribution alpha from 1.0 to 0.5 (ablation) | MemoryLLM-1B · BoolQ | accuracy: 0.6205 -> 0.6131 | No clear effect |
| reduce FFN contribution alpha from 1.0 to 0.5 (ablation) | MemoryLLM-1B · PIQA | accuracy: 0.7018 -> 0.6801 | No clear effect |
| reduce FFN contribution (alpha) in MemoryLLM versus conventional base (ablation) | MemoryLLM vs dense base · Wikitext-2, SciQ, PIQA, HellaSwag | performance change | Improved |
| add MemoryLLM/Flex-MemoryLLM design instead of pruning a dense model (ablation) | dense base 1B and 750M · C4 | perplexity | Improved |
| use independent layer norm per FFN | MemoryLLM · training convergence | convergence | Improved |
Closest papers by meaning
- An Evolved Universal Transformer Memory (ICLR 2025 · similarity 0.87)
- Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory (AAAI 2024 · similarity 0.87)
- Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding (NeurIPS 2025 · similarity 0.86)
- Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling (NeurIPS 2024 · similarity 0.86)
- Understanding Factual Recall in Transformers via Associative Memories (ICLR 2025 · similarity 0.85)
- On-the-Fly Adaptive Distillation of Transformer to Dual-State Linear Attention for Long-Context LLM Serving (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
