Research map / Label-efficient and robust learning / Recognition and long-tail
Generalization through Memorization: Nearest Neighbor Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add kNN retrieval over the training set datastore (kNN-LM interpolation) | Baevski & Auli (2019) Transformer LM · WIKITEXT-103 | test perplexity: 18.65 -> 16.12 | Helped |
| add continuous cache on top of kNN-LM | kNN-LM · WIKITEXT-103 | test perplexity: 16.12 -> 15.79 | Helped |
| add continuous cache alone | Baevski & Auli (2019) LM · WIKITEXT-103 | test perplexity: 18.65 -> 18.27 | Helped |
| add kNN retrieval over BOOKS training data as datastore | BOOKS LM · BOOKS | test perplexity: 11.89 -> 10.89 | Helped |
| use a 3B-token datastore with a 100M-token trained LM instead of training on 3B tokens | WIKI-100M LM · WIKI-3B | test perplexity: 19.59 -> 13.73 (vanilla 3B-trained LM: 15.17) | Helped |
| add in-domain BOOKS datastore to WIKI-3B-trained model (domain adaptation) | WIKI-3B LM · BOOKS domain adaptation | test perplexity: 34.84 -> 20.47 | Helped |
| use FFN input after layer norm (final layer) as key representation instead of model output (ablation) | kNN-LM · WIKITEXT-103 validation | dev perplexity: 17.07 -> 16.06 | Helped |
| use normalized (post layer norm) representations as keys (ablation) | kNN-LM · WIKITEXT-103 validation | dev perplexity: 17.06 (before LN) -> 16.06 (after LN) for FFN input | Helped |
| increase number of retrieved neighbors k from 1 to 1024 (ablation) | kNN-LM · WIKITEXT-103 validation | perplexity: monotonic improvement; k=8 already reaches state of the art | Helped |
| compute squared L2 distances with full-precision keys instead of quantized FAISS distances (ablation) | kNN-LM · WIKITEXT-103 | perplexity: 16.5 -> 16.06 | Helped |
| interpolate with n-gram LM instead of kNN retrieval (ablation) | Transformer LM · WIKITEXT-103 validation | perplexity: about 0.2 improvement only, versus 1.9 for kNN-LM | Hurt |
| train LM with no dropout to memorize training data and interpolate it with original LM (ablation) | Transformer LM · WIKITEXT-103 validation | perplexity: best validation 28.59 alone; interpolation improves only by 0.1 | Hurt |
Closest papers by meaning
- Test-Time Training on Nearest Neighbors for Large Language Models (ICLR 2024 · similarity 0.90)
- Mitigating Memorization in Language Models (ICLR 2025 · similarity 0.88)
- Once Read is Enough: Domain-specific Pretraining-free Language Models with Cluster-guided Sparse Experts for Long-tail Domain Knowledge (NeurIPS 2024 · similarity 0.87)
- Generalization v.s. Memorization: Tracing Language Models’ Capabilities Back to Pretraining Data (ICLR 2025 · similarity 0.87)
- Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models (NeurIPS 2024 · similarity 0.87)
- DEPT: Decoupled Embeddings for Pre-training Language Models (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.