Research map / Architectures and efficiency / RNNs and normalization
Improving Neural Language Models with a Continuous Cache
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a neural cache to the LSTM | LSTM on Penn Tree Bank · Penn Tree Bank test | perplexity: 82.3 -> 72.1 | Helped |
| Add a neural cache of size 2,000 | LSTM on wikitext2 · wikitext2 test | perplexity: 99.3 -> 68.9 | Helped |
| Increase cache size from 100 to 2,000 (ablation) | Neural cache model on wikitext2 · wikitext2 test | perplexity: 81.6 -> 68.9 | Helped |
| Add a neural cache of size 2,000 | LSTM on wikitext103 · wikitext103 test | perplexity: 48.7 -> 40.8 | Helped |
| Increase cache size from 100 to 2,000 (ablation) | Neural cache model on wikitext103 · wikitext103 test | perplexity: 44.8 -> 40.8 | Helped |
| Add a neural cache | LSTM-1024 on text8 · text8 test | perplexity: 121.8 -> 99.9 | Helped |
| Add a neural cache | LSTM-1024 on LAMBADA · LAMBADA Dev | perplexity: 4088 -> 138 | Helped |
| Add a neural cache | LSTM-1024 on LAMBADA · LAMBADA Ctrl | perplexity: 94 -> 129 | Hurt |
| Use linear interpolation instead of global normalization (ablation) | Neural cache model on Penn Tree Bank · Penn Tree Bank validation | perplexity: 74.9 -> 74.6 | Helped |
| Use linear interpolation instead of global normalization (ablation) | Neural cache model on wikitext2 · wikitext2 validation | perplexity: 73.5 -> 72.1 | Helped |
| Add a neural cache | LSTM on LAMBADA · LAMBADA | perplexity: 4088 -> 138; 94 -> 129 | Mixed |
Closest papers by meaning
- Language Model Memory and Memory Models for Language (NeurIPS 2026 · similarity 0.87)
- Simple linear attention language models balance the recall-throughput tradeoff (ICML 2024 · similarity 0.87)
- Memory Layers at Scale (ICML 2025 · similarity 0.86)
- Peripheral Memory for LLMs: Integration of Sequential Memory Banks with Adaptive Querying (ICML 2025 · similarity 0.86)
- M+: Extending MemoryLLM with Scalable Long-Term Memory (ICML 2025 · similarity 0.86)
- Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memory (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.