Research map / Continual learning / Continual fine-tuning of LLMs
Skill Neologisms: Towards Skill-based Continual Learning
Antonin Berthon, Nicolás Astorga, Mihaela van der Schaar
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| insert skill neologism tokens in prompt (vs LoRA) for composing the new skill with held-out skills | Qwen2.5-0.5B on digit-sequence skills · OOD 2-combinations, SHIFT | accuracy: 0.16 -> 0.75 (LoRA vs skill neologisms) | Improved |
| insert soft tokens inside the prompt as a neologism rather than as a prefix (prompt tuning) | skill neologisms · OOD 2-combinations, INV-POL | accuracy: 0.57 -> 0.84 | Improved |
| compose independently learned SHIFT and INV-POL neologisms zero-shot instead of in-context examples | skill neologisms · digit-sequence two-skill composition | accuracy | Improved |
| increase neologism length beyond l=20 (ablation) | skill neologisms · OOD 2-combinations with held-out ADD | accuracy | Hurt |
| use very short neologism (l=1) (ablation) | skill neologisms · digit-sequence skills | accuracy | Hurt |
| train with higher k_max (more complex compositions) in the training set (ablation) | skill neologisms, INV-POL · OOD 2-skill compositions | accuracy: .36 -> .85 (k_max 1 -> 3) | Improved |
| train with higher k_max (ablation) | skill neologisms, SHIFT · OOD 2-skill compositions | accuracy: .46 -> .72 (k_max 1 -> 3); .72 at k_max 2 and 3 | Improved |
| initialize from mean pretrained skill embedding instead of random (ablation) | skill neologisms, INV-POL · OOD 2-skill compositions | accuracy: .63 -> .85 | Improved |
| initialize from mean pretrained skill embedding instead of random (ablation) | skill neologisms, SHIFT · OOD 2-skill compositions | accuracy: .70 -> .72 | No clear effect |
| add noise to skill labels in training data (20%+) (ablation) | skill neologisms, INV-POL · held-out ADD, 2-op and 3-op | accuracy | Hurt |
| add noise to skill labels (up to about 40%) (ablation) | skill neologisms, INV-POL · held-out ADD, 1-op | accuracy | No clear effect |
| combine independently trained skill neologisms (modus ponens + statistical syllogism) | Llama3.2-3B-Instruct · Skill-Mix, both skills | accuracy | Improved |
Closest papers by meaning
- From f(x) and g(x) to f(g(x)): LLMs Learn New Skills in RL by Composing Old Ones (ICLR 2026 · similarity 0.89)
- Skill-Pro: Learning Reusable Skills from Experience via Non-Parametric PPO for LLM Agents (ICML 2026 · similarity 0.88)
- SkillOpt: Executive Strategy for Self-Evolving Agent Skills (NeurIPS 2026 · similarity 0.87)
- Hypothesis, Verification, and Induction: Grounding Large Language Models with Self-Driven Skill Learning (AAAI 2024 · similarity 0.87)
- Separating the 'what' and 'how' of compositional computation to enable reuse and continual learning (NeurIPS 2025 · similarity 0.87)
- Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
