Research map / Continual learning / Continual fine-tuning of LLMs
LAMOL: LAnguage MOdeling for Lifelong Language Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replay generated pseudo-samples from previous tasks during sequential training | LAMOL · SST, QA-SRL, WOZ (six task orders) | average score: 39.3 -> 79.7 | Helped |
| Increase the pseudo-sample ratio from γ=0.05 to γ=0.2 (ablation) | LAMOL-GEN · SST, QA-SRL, WOZ (six task orders) | average score: 74.9 -> 79.7 | Helped |
| Use task-specific generation tokens instead of a shared GEN token (ablation) | LAMOL at γ=0.05 · SST, QA-SRL, WOZ (six task orders) | average score: 74.9 -> 76.0 | Helped |
| Use task-specific generation tokens instead of a shared GEN token (ablation) | LAMOL at γ=0.2 · SST, QA-SRL, WOZ (six task orders) | average score: 79.7 -> 79.5 | No clear effect |
| Add task-specific tokens when no pseudo-samples are generated (γ=0) (ablation) | LAMOL · SST, QA-SRL, WOZ (six task orders) | average score: 44.8 -> 42.9 | Hurt |
| Increase the pseudo-sample ratio from γ=0.05 to γ=0.2 (ablation) | LAMOL-TASK · Five decaNLP tasks | average score: 71.5 -> 74.3 | Helped |
| Use task-specific tokens instead of a shared GEN token (ablation) | LAMOL at γ=0.05 · Five decaNLP tasks | average score: 69.6 -> 71.5 | Helped |
| Use task-specific tokens instead of a shared GEN token (ablation) | LAMOL at γ=0.2 · Five decaNLP tasks | average score: 73.1 -> 74.3 | Helped |
| Replay generated pseudo-samples rather than real previous-task samples (ablation) | LAMOL · Five decaNLP tasks | average score: 74.5 -> 69.6 | Hurt |
| Replay pseudo-samples from previous tasks | LAMOL-TASK · Text classification (four task orders) | average EM: 74.2 -> 76.5 | Helped |
| Increase the pseudo-sample ratio (ablation) | LAMOL · WikiSQL, SST, QA-SRL, WOZ task stream | overall performance | No clear effect |
Closest papers by meaning
- EvoLM: In Search of Lost Language Model Training Dynamics (NeurIPS 2025 · similarity 0.86)
- Stress-Testing Long-Context Language Models with Lifelong ICL and Task Haystack (NeurIPS 2024 · similarity 0.85)
- Scalable Language Model with Generalized Continual Learning (ICLR 2024 · similarity 0.83)
- Embracing Language Inclusivity and Diversity in CLIP through Continual Language Learning (AAAI 2024 · similarity 0.83)
- COPAL: Continual Pruning in Large Language Generative Models (ICML 2024 · similarity 0.83)
- Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.