AI Paper Atlas

Research map / Large language models / Retrieval and knowledge

Pre-training Limited Memory Language Models with Internal and External Knowledge

Linxi Zhao, Sofian Zalouk, Christian Belardi, Justin Lovelace, Jin Peng Zhou, Ryan Thomas Noonan, Dongyoung Go, Kilian Q Weinberger, Yoav Artzi, Jennifer J. Sun

ICLR 2026 · Large language models › Retrieval and knowledge · approach: Knowledge editing in LLMs · 14 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
externalize facts to a database with masked-loss lookup pretrainingLLaMA2-382M · FactScoreFactScore: 14.0 -> 31.9Improved
externalize facts with lookup pretrainingLLaMA2-382M · T-RExEM: 52.0 -> 58.1Improved
externalize facts with lookup pretrainingGPT2-124M · PopQA long-tailaccuracy: 18.51 -> 49.89Improved
LMLM instead of Standard (validation perplexity)GPT2/LLaMA2 small models · Wikipedia validationperplexity (Dynamic): average reduction 1.98Improved
LMLM instead of Standard (NLU)GPT2/LLaMA2 small models · five NLU tasksaccuracyNo clear effect
disable external database at inference (ablation)LMLM LLaMA2-382M · FactScoreFactScore: 31.9 -> 12.8Hurt
disable external database at inference (ablation)LMLM LLaMA2-382M · T-RExEM: 58.1 -> 38.5Hurt
LMLM vs RAG with top-4 Wikipedia articlesGPT2-355M · T-RExEM: 75.8 (RAG) vs 58.7 (LMLM)Hurt
LMLM vs RAGGPT2-355M · PopQAaccuracy: 37.5 (RAG) -> 52.0 (LMLM)Improved
increase knowledge offloading ratio toward 100% (ablation)LMLM · perplexity and factual precisionperplexity, factual precisionImproved
mask retrieved values from loss, delete forget-set entries (no training)LMLM LLaMA-3.2-1B · TOFU forget 5%forget quality / utilityImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy