Research map / Large language models / Retrieval and knowledge
Self-Updatable Large Language Models by Integrating Context into Model Parameters
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SELF-PARAM, which injects contexts into LLM parameters with zero extra storage by minimizing KL divergence between a context-conditioned original model and a context-free target model on generated QA pairs plus SlimPajama text.
Problem
Rapidly and frequently integrating small-scale experiences/contexts into LLM parameters with both high immediate efficacy and long-term retention, without external storage modules.
Approach
Train the target model (no context) to match the next-token distributions of the original model conditioned on the context, using a KL objective over diverse LLM-generated QA pairs about the context plus unrelated SlimPajama sentences, instead of next-word-prediction loss on the raw context.
Key results
- Single context injection on Mistral-7B: SELF-PARAM reaches 0.3705 QA-F1 vs 0.3246 for FT (S) and 0.2433 for FT (C), against 0.5461 upper bound with context.
- Batch injection (500 contexts, Llama3-8B): SELF-PARAM 0.4221 QA-F1 vs best zero-storage-free-competitor BM25 0.2862 and DPR 0.2310, with zero additional storage.
- Conversational recommendation on REDIAL: SELF-PARAM Recall@1 0.0337 (r1) vs Base 0.0316, while FT (C) and FT (S) drop to 0.0000.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SELF-PARAM instead of fine-tuning on the target sentence set | OpenLLaMA-3B-v2 · Single Context Injection | QA-F1: 0.2742 -> 0.4927 | Helped |
| Use SELF-PARAM instead of fine-tuning on the target sentence set | Mistral-7B · Single Context Injection | QA-F1: 0.3246 -> 0.3705 | Helped |
| Use SELF-PARAM instead of fine-tuning on the target sentence set | Llama3-8B · Single Context Injection | QA-F1: 0.3115 -> 0.4213 | Helped |
| Use SELF-PARAM instead of the strongest listed baseline | OpenLLaMA-3B-v2 and Mistral-7B · Batch Context Injection, 100 contexts | QA-F1: 0.2795 -> 0.5082; 0.3175 -> 0.4521 | Helped |
| Use SELF-PARAM instead of the strongest listed baseline | OpenLLaMA-3B-v2 · Batch Context Injection, 500 contexts | QA-F1: 0.2528 -> 0.5048 | Helped |
| Use SELF-PARAM instead of the strongest listed baseline | Mistral-7B and Llama3-8B · Batch Context Injection, 500 contexts | QA-F1: 0.3459 -> 0.4384; 0.2862 -> 0.4221 | Helped |
| Inject conversations with SELF-PARAM rather than the strongest baseline | Mistral-7B-Instruct-v0.2 · INSPIRED and REDIAL conversational recommendation | Recall@1: 0.0356 -> 0.0395; 0.0337 -> 0.0360 | Helped |
| Use the corresponding instruct model instead of gpt-4o-mini to generate the target sentence set (ablation) | SELF-PARAM · Batch Injection, 100 contexts, Mistral-7B and Llama3-8B | QA-F1: 0.4521 -> 0.4502; 0.4368 -> 0.4341 | Hurt |
| Use the corresponding instruct model instead of gpt-4o-mini to generate the target sentence set (ablation) | SELF-PARAM · Batch Injection, 500 contexts, Mistral-7B and Llama3-8B | QA-F1: 0.4384 -> 0.4836; 0.4221 -> 0.4464 | Helped |
| Use 10 QA pairs for 5 epochs instead of 50 QA pairs for 1 epoch (ablation) | SELF-PARAM, OpenLLaMA-3B-v2 · Batch Injection, 100 and 500 contexts | QA-F1: 0.5082 -> 0.4203; 0.5048 -> 0.4203 | Hurt |
| Use 10 QA pairs for 5 epochs instead of 50 QA pairs for 1 epoch (ablation) | SELF-PARAM, Mistral-7B · Batch Injection, 100 and 500 contexts | QA-F1: 0.4521 -> 0.3891; 0.4384 -> 0.3813 | Hurt |
| Remove SlimPajama training sentences (ablation) | SELF-PARAM · Sequential Injection | QA-F1: around 0.5; less than 0.30 | Hurt |
Surprising findings
- Fine-tuning directly on the context with next-word prediction (FT (C)) barely injects knowledge, and fine-tuning on conversations collapses recommendation recall to zero.
- Using a same-family instruct model instead of gpt-4o-mini to generate the target sentence set gives comparable results, sometimes better (e.g., 500 contexts on Mistral-7B: 0.4836 vs 0.4384).
Limitations
- Roughly doubles computational cost relative to fine-tuning baselines due to QA generation and base-logit computation.
- Sequential injection retention decays from about 0.5 to about 0.33 QA-F1 after 20 contexts, and experiments are limited to small context counts and a few models.
- Risk of hallucination when the model answers without the context; mitigated only by filtering prompts that require prior context.
Experimental setup
Datasets: PwC SlimPajama INSPIRED REDIAL
Benchmarks: PwC INSPIRED REDIAL
Metrics: QA-F1 Recall@1
Baselines: Base FT (C) FT (S) MemoryLLM-8B InfLLM DPR BM25 RAPTOR
Builds on: LoRA MEND MEMIT RAPTOR MemoryLLM
Closest papers by meaning
- MEMORYLLM: Towards Self-Updatable Large Language Models (ICML 2024 · similarity 0.91)
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.90)
- Training with “Paraphrasing the Original Text” Teaches LLM to Better Retrieve in Long-Context Tasks (AAAI 2025 · similarity 0.89)
- WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models (NeurIPS 2024 · similarity 0.89)
- Physics of Language Models: Part 3.1, Knowledge Storage and Extraction (ICML 2024 · similarity 0.89)
- Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.