AI Paper Atlas

Research map / Continual learning / Continual fine-tuning of LLMs

Improving Language Plasticity via Pretraining with Active Forgetting

Yihong Chen, Kelly Marchisio, Roberta Raileanu, David Ifeoluwa Adelani, Pontus Stenetorp, Sebastian Riedel, Mikel Artetxe

NeurIPS 2023 · Continual learning › Continual fine-tuning of LLMs · approach: Continual pretraining of language models · 2 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Reset token embeddings periodically during pretrainingRoBERTa-base · XNLI, 5M-token language adaptationaccuracy: 53.3 -> 62.7; +21.2%Improved
Reset token embeddings periodically during pretrainingRoBERTa-base · MLQA, 5M-token language adaptationF1: 34.3 -> 43.4; +33.8%Improved
Reset token embeddings periodically during pretrainingRoBERTa-base · XQuAD, 5M-token language adaptationF1: 36.1 -> 49.0; +60.9%Improved
Reset token embeddings periodically during pretrainingRoBERTa-base · English XNLIaccuracy: 86.1 -> 85.1; -1.2%Hurt
Reset token embeddings periodically during pretrainingRoBERTa-base · XNLI Thai after multilingual pretrainingaccuracy: 49.4 -> 55.0Improved
Reset token embeddings periodically during pretrainingRoBERTa-base · XNLI Thai after English monolingual pretrainingaccuracy: 35.3 -> 59.7Improved
Reset token embeddings periodically during pretrainingRoBERTa-base · Arabic NLI with 5M unlabeled tokensaccuracy: 41.2 -> 59.7Improved
Reset token embeddings periodically during pretrainingRoBERTa-base · XNLI during early language adaptationaccuracy: 37.2 -> 57.8Improved
Adapt the full model on the task instead of separately adapting embeddings and task bodyStandard multilingual RoBERTa-base · XNLI Thaiaccuracy: 49.4 -> 60.0Improved
Use forgetting pretraining with standard labeled adaptation instead of standard pretraining with standard adaptationRoBERTa-base · Arabic NLIaccuracy: 32.8 -> 34.2Improved
Set the forgetting interval to K=100 instead of K=1000 (ablation)Active forgetting pretraining · Pretraining lossloss: around 11Hurt
Set the forgetting interval to K=5000 instead of K=1000 (ablation)Active forgetting pretraining · Pretraining stabilitydivergenceHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy