AI Paper Atlas

Research map / Continual learning / Continual fine-tuning of LLMs

Memory as a Markov Matrix: Sample Efficient Knowledge Expansion via Token-to-Dictionary Mapping

Kaustubh Pethkar, Ziyang Xiong, Zuofeng Shang, Yingcong Li

ICML 2026 · Continual learning › Continual fine-tuning of LLMs · approach: Continual pretraining of language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
embedding tuning (train only new token embedding) instead of full fine-tuningEmbedding tuning (ET) · Synthetic vocabulary, Qwen2.5-7B (N=1000)WikiText forgetting (loss increase): +7.37 -> .00Improved
embedding tuning instead of LoRAEmbedding tuning (ET) · Synthetic vocabulary, Llama3.1-8B (N=1000)test loss: 7.63 -> 3.79Improved
embedding tuning instead of prompt tuningEmbedding tuning (ET) · Synthetic vocabulary, Llama3.2-1B (N=1000)test loss: 6.37 -> 3.50Improved
prompt tuning beats embedding tuning on Phi-3.5 Mini (data generator model)Embedding tuning (ET) · Synthetic vocabulary, Phi-3.5 Mini (N=1000)test loss: 1.87 vs 2.75Hurt
embedding tuning instead of FFT on multiplication tokenEmbedding tuning (ET) · Llama-3.2-3B-Instruct arithmetic <spec> operatoraccuracy of a<spec>b: 81.4 vs 63.5 for a*bImproved
full fine-tuning on new operator causes addition forgettingFull fine-tuning · Llama-3.2-3B-Instruct addition a+baccuracy: 100 -> 0Hurt
embedding tuning keeps addition accuracyEmbedding tuning (ET) · Llama-3.2-3B-Instruct addition a+baccuracy: 100 -> 100Improved
ensemble of multiplication phrasings explains <spec> accuracy (ablation)Base model prompts · Llama-3.2-3B-Instruct baseaccuracy: 63.5 -> 73.2Improved
embedding tuning instead of FFT for Arabic expansionEmbedding tuning (ET) · Qwen2.5-3B English-to-Arabictarget-language loss: 7.95 -> 2.82Improved
embedding tuning instead of FFT for Arabic expansionEmbedding tuning (ET) · Qwen2.5-3B English-to-ArabicEnglish forgetting: +11.95 -> 0.00Improved
embedding tuning instead of LoRA for German expansionEmbedding tuning (ET) · Qwen2.5-3B English-to-Germantarget-language loss: 3.29 -> 3.27Improved
embedding tuning outperforms full fine-tuning in low-data regimeEmbedding tuning (ET) · Llama-3.2-3B-Instruct multiplicationaccuracyImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy