Research map / Large language models / Retrieval and knowledge
MLP Memory: A Retriever-Pretrained Memory for Large Language Models
Rubin Wei, Jiaqi Cao, Jiarui Wang, Jushi Kai, Qipeng Guo, Bowen Zhou, Zhouhan Lin
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add MLP Memory (interpolated) to the base LLM | Mistral-7B-v0.3 · Five QA benchmarks | average score: 32.12 -> 36.06 | Improved |
| add MLP Memory to the base LLM | Llama2-7B · Five QA benchmarks | average score: 32.81 -> 35.38 | Improved |
| continued pretraining or LoRA on Wikipedia instead of MLP Memory | Llama2-7B · Five QA benchmarks | average score: 32.81 -> 29.66 (CPT), 31.34 (LoRA) | Hurt |
| use RAG (top-5 documents) | Mistral-7B-v0.3 · WebQA | exact match: 29.28 -> 24.90 | Hurt |
| add MLP Memory | Mistral-7B-v0.3 · Nine general NLP tasks | average accuracy: 67.86 -> 73.07 | Improved |
| add MLP Memory | Mistral-7B-v0.3 · HaluEval dialogue, QA, summarization | accuracy: +9.68, +10.08, +2.14 | Improved |
| apply CPT to the base model | Mistral-7B-v0.3 · HaluEval dialogue | accuracy: 57.18 -> 51.68 | Hurt |
| use MLP Memory architecture instead of continued training of decoder-only GPT-2 | GPT-2 family · WikiText-103 scaling | power-law exponent: -0.143 -> -0.168 | Improved |
| use MLP Memory architecture instead of continued training of decoder-only GPT-2 | GPT-2 family · Web dataset scaling | power-law exponent: -0.216 -> -0.268 | Improved |
| vary KL/CE loss weight alpha away from 0.4 (to 0.0 or 1.0) (ablation) | MLP Memory training · WikiText-103 (GPT-2 small/medium/large) | perplexity | Hurt |
| attach MLP Memory at about 70% network depth instead of the final layer (ablation) | MLP Memory · WikiText-103 (GPT-2 small/medium/large) | perplexity | Improved |
| raise interpolation weight lambda to 0.9 (ablation) | MLP Memory · HaluEval dialogue | accuracy: 64.07 (lambda 0.45) -> 49.71 (lambda 0.90) | Hurt |
Closest papers by meaning
- From RAG to Memory: Non-Parametric Continual Learning for Large Language Models (ICML 2025 · similarity 0.88)
- Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memory (NeurIPS 2026 · similarity 0.88)
- Memory Decoder: A Pretrained, Plug-and-Play Memory for Large Language Models (NeurIPS 2025 · similarity 0.88)
- Frozen Memory Is Not Enough: Rethinking External Memory as Extraction (NeurIPS 2026 · similarity 0.88)
- RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval (NeurIPS 2025 · similarity 0.88)
- Mask-Enhanced Autoregressive Prediction: Pay Less Attention to Learn More (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
