Research map / Architectures and efficiency / Attention and transformers
Integrated Episodic and Semantic Memory via Modulating Transformer FeedForward Layers
Yiqun Yao, Xiang Li, Xin Jiang, Xuezhi Fang, Naitong Yu, Siwei Dong, Wenjia Ma, Jing Li, Aixin Sun, Yequan Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add hypernetwork-based memory (HyperMem) over Generative Adapter | HyperMem · Raw text retrieval (random access) | F1: 0.297 -> 0.654 | Improved |
| use HyperMem instead of MemoryLLM-8B-1024 with similar memory size | HyperMem · Raw text retrieval (from head) | F1: 0.749 -> 0.876 | Improved |
| HyperMem vs MemoryLLM-8B-12800 which has far larger storage | HyperMem · Raw text retrieval (from head) | F1: 0.901 -> 0.876 | Mixed |
| HyperMem vs Generative Adapter on LongBench | HyperMem · LongBench HotpotQA | F1: 0.385 -> 0.420 | Improved |
| HyperMem vs Generative Adapter on MultiFieldQA-EN | HyperMem · LongBench MultiFieldQA-EN | F1: 0.186 -> 0.213 | Improved |
| HyperMem vs full-context Llama3.1-8B on HotpotQA | HyperMem · LongBench HotpotQA | F1: 0.431 -> 0.420 | Hurt |
| HyperMem vs Generative Adapter on MSC personalization | HyperMem · MemGPT-MSC | F1: 40.2 -> 43.0 | Improved |
| modulate parameters with memory injection on unrelated text | HyperMem · FineWebEdu val perplexity | perplexity: 7.66 -> 7.77 | No clear effect |
| modulate attention layers only or both attention and FFN instead of FFN only (ablation) | HyperMem · Training loss (early) | loss | Hurt |
| replace LoRA encoder with a randomly initialized shared Transformer layer (decouple layer) (ablation) | HyperMem · Training loss (early) | loss | Hurt |
| remove the hypernetwork and use memory as extended KV cache (no hyper) (ablation) | HyperMem · Episodic memory loss (early training) | loss | No clear effect |
| increase LoRA rank from 16 to 64 (ablation) | HyperMem · Associative memory training loss | loss: delta loss = 0.1 | Improved |
Closest papers by meaning
- Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory (AAAI 2024 · similarity 0.89)
- REMem: Reasoning with Episodic Memory in Language Agent (ICLR 2026 · similarity 0.88)
- Human-inspired Episodic Memory for Infinite Context LLMs (ICLR 2025 · similarity 0.88)
- Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers (NeurIPS 2025 · similarity 0.87)
- RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling (NeurIPS 2025 · similarity 0.87)
- Episodic Memories Generation and Evaluation Benchmark for Large Language Models (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
