AI Paper Atlas

Research map / Continual learning / Continual fine-tuning of LLMs

Updating Parametric Knowledge with Context Distillation Retains Post-Training Capabilities

Shankar Padmanabhan, Mustafa Omer Gul, Tanya Goyal

ICML 2026 · Continual learning › Continual fine-tuning of LLMs · approach: Continual pretraining of language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use DiSC context distillation instead of standard fine-tuning (capability-preserving LR)Qwen2.5-7B-Instruct · KUPaccuracy (%): 36.3 -> 44.18Improved
use DiSC instead of FT at unconstrained learning rateQwen2.5-7B-Instruct · KUPMATH-Hard / IFEval: IFEval 65.83 -> 77.70; Math 24.47 -> 44.34Improved
use DiSC instead of CD basepost-trained LMs · KUP, Llama3.1-8B / Qwen2.5-7B / Qwen3-8Baccuracy (%): 35.80 -> 44.88 (Llama)Improved
use DiSC instead of FTpost-trained LMs · BioASQaccuracy (%): 73.08 -> 81.73 (Qwen2.5-7B)Improved
add KL regularization to fine-tuning (ablation)Qwen2.5-7B-Instruct · KUPIFEval / MATH forgetting: IFEval 65.83 -> 69.78; Math 24.47 -> 27.57No clear effect
add TALR token-adaptive loss to fine-tuning (ablation)Qwen2.5-7B-Instruct · KUPMATH forgetting: 24.47 -> 26.51No clear effect
add LoRA to fine-tuning (ablation)Qwen2.5-7B-Instruct · KUPKUP accuracy and IFEval: KUP 40.9 -> 35.6; IFEval 65.83 -> 75.90Mixed
add on-policy rephrases to fine-tuning (ablation)Qwen2.5-7B-Instruct and Llama3.1-8B · KUPadaptation accuracy: 40.90 -> 42.90 (Qwen 7B)Mixed
add replay data (GSM8K gold) to fine-tuning (ablation)Qwen2.5-7B-Instruct · KUPMATH: 24.47 -> 2.95Hurt
add replay data (AlpacaEval gold) to fine-tuning (ablation)Qwen2.5-7B-Instruct · KUPIFEval: 65.83 -> 48.44Hurt
use DiSC instead of FT as learning rate increases (ablation)Qwen2.5-7B-Instruct · KUPIFEval drop: -14.87 -> -8.4Improved
fine-tune on KUP (standard FT)post-trained LMs · IFEval, Math, Code vs BBH, GPQA, MuSRdegradationHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy