Research map / Architectures and efficiency / Attention and transformers
Dual Process Learning: Controlling Use of In-Context vs. In-Weights Strategies with Weight Forgetting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies structural in-context learning (generalizing to random novel tokens) in LMs, shows it is transient in training, and proposes active and temporary forgetting of embeddings to maintain it and induce dual-process (ICL + IWL) behavior, plus probabilistic temporary forgetting for fine-tuning GPT-2.
Problem
Language models fail on unseen or rarely seen tokens because in-context learning ability degrades during training, so they cannot generalize structurally to novel tokens while still memorizing common ones.
Approach
Extends active forgetting (periodically re-initializing the embedding matrix) with temporary forgetting, applying resets only for the first N steps so the model keeps structural ICL for tail/unseen tokens while memorizing head tokens; a probabilistic variant applies random-embedding replacement during fine-tuning of pretrained LMs.
Key results
- Probabilistic temporary forgetting raised GPT-2 large unseen-token syllogism accuracy from 0.02 to 0.927 while baseline accuracy stayed about 0.92 (0.933 to 0.923).
- Vanilla training makes structural ICL transient: MultiBERT Random Token probe accuracy peaks early and drops to chance by end of training.
- Temporary forgetting enables dual process learning, maintaining structural ICL for random tokens while keeping IWL on head tokens across Zipf alpha >= 1.0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use probabilistic temporary forgetting during fine-tuning instead of vanilla fine-tuning | GPT-2 large · Syllogism, unseen tokens | accuracy: 0.02 to 0.927 | Helped |
| Use probabilistic temporary forgetting during fine-tuning | GPT-2 large · Syllogism, baseline conditions | accuracy: 0.933 to 0.923 | No clear effect |
| Increase weight decay from 0.01 to 0.1 (ablation) | Synthetic POS model · Synthetic POS task, structural ICL | accuracy | No clear effect |
| Use active forgetting instead of vanilla training | Synthetic masked language model · Random Token dataset across tested distributions | accuracy | Helped |
| Use active forgetting instead of vanilla training | Synthetic masked language model · Synthetic POS task, in-weights learning | accuracy | Hurt |
| Use temporary forgetting instead of vanilla training | Synthetic masked language model · Tail and unseen tokens | accuracy | Helped |
| Use temporary forgetting instead of active forgetting | Synthetic masked language model · Head tokens, in-weights learning | accuracy | Helped |
| Vary temporary forgetting duration N (ablation) | Synthetic masked language model · Zipf distribution, α=1.5 | accuracy | Mixed |
| Increase ambiguity ε (ablation) | Synthetic masked language model · Synthetic POS task, unambiguous tokens | accuracy | Mixed |
| Vary vocabulary size (ablation) | Synthetic masked language model · Synthetic POS task, structural ICL | accuracy | Mixed |
| Use temporary forgetting instead of vanilla training | Autoregressive transformer · Few-shot task, Zipf distributions α=1.0001, 2, 3 | accuracy | Helped |
| Use temporary forgetting instead of vanilla fine-tuning | GPT-2 large · Syllogism, unseen tokens | accuracy: 0.02 to 0.927 | Helped |
Surprising findings
- Prior L2-regularization method for persistent conditional ICL does not maintain structural ICL; undertrained and random tokens still perform poorly.
- Epsilon (label ambiguity) must be greater than zero for an in-context solution to emerge at all.
Limitations
- Probabilistic temporary forgetting is only a proof-of-concept on GPT-2 large with one syllogism task.
- Synthetic experiments use small toy models and a fixed task design; generalization to large-scale pretraining is untested.
- Hyperparameters (k, N) were chosen by limited search and preliminary experiments.
Experimental setup
Datasets: Penn Treebank-3 Wikitext BookCorpus English Wikipedia English UD Treebank OntoNotes-v5
Benchmarks: Head/Tail/Head Switch/Tail Switch/Random Token POS probing datasets syllogism task with unseen tokens
Metrics: accuracy probe accuracy unseen token accuracy
Baselines: vanilla training active forgetting L2 regularization
Builds on: active forgetting L2 regularization for ICL persistence MultiBERTs
Closest papers by meaning
- Dual Operating Modes of In-Context Learning (ICML 2024 · similarity 0.91)
- Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Ability (ICLR 2024 · similarity 0.90)
- Capturing In-Context Learning Dynamics with Task Operators (NeurIPS 2026 · similarity 0.89)
- Toward Understanding In-context vs. In-weight Learning (ICLR 2025 · similarity 0.89)
- In-Context Language Learning: Architectures and Algorithms (ICML 2024 · similarity 0.89)
- Position: Do pretrained Transformers Learn In-Context by Gradient Descent? (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.