Research map / Large language models / Instruction and fine-tuning
Neutral residues: revisiting adapters for model extension
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes "neutral residues," gated parallel adapters trained with an l1 sparsity loss on original-domain data so new residual blocks output near-zero there, to extend LLMs to new languages with less forgetting of English.
Problem
Extending a pretrained LLM to a new language/domain adds knowledge but causes forgetting of the original domain; fine-tuning and LoRA add no formal capacity and trade off learning vs. forgetting.
Approach
Parallel FFN adapters with a ReLU block gate trained by an l1 local loss on original-distribution data, so each new residual block outputs near-zero on the original domain, combined with low-variance initialization and mixed data (p=0.1 English Wikipedia).
Key results
- On Gemma-2B extended to French, neutral residues reach a task average of 53.3 on English (forgetting) and 48.2 on French (learning), vs. 49.8/49.9 for finetuning and 51.2/44.9 for LoRA.
- Neutral residues outperform vanilla adapters on the learning-vs-forgetting trade-off across Danish, French, Hungarian and Slovak, and stay stable across a learning-rate range of one order of magnitude.
- Ablation: the l1-regularized ReLU gate improves French task average from 42.4 (no loss) to 43.6, and alpha=0.01 gives the best trade-off.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add a ReLU block gate and L1 local loss instead of a vanilla adapter (ablation) | EN-LM-1B extension · French | French task average: 41.3 -> 43.6 | Helped |
| add L1 supervision to a ReLU-gated adapter (ablation) | EN-LM-1B extension · French | French task average: 42.8 -> 43.6 | Helped |
| remove the block gate while retaining L1 regularization (ablation) | EN-LM-1B extension · French | French task average: 43.6 -> 42.4 | Hurt |
| replace ReLU plus L1 with sigmoid plus cross-entropy gating (ablation) | EN-LM-1B extension · French | French task average: 43.6 -> 42.6 | Hurt |
| use low-variance initialization instead of He initialization (ablation) | Neutral residues · French | French task average: 41.6 -> 43.6 | Helped |
| set the L1 loss coefficient to 0.01 rather than 0 (ablation) | Neutral residues · French | French task average: 42.8 -> 43.6 | Helped |
| increase original-domain data proportion from p=0 to p=0.1 (ablation) | EN-LM-1B extension with vanilla adapters · English and French validation | perplexity: English: 0.720 -> 0.687; French: 0.810 -> 0.812 | Mixed |
| extend Gemma-2B with neutral residues rather than vanilla adapters | Gemma-2B · Danish target learning | task average: 42.3 -> 42.9 | Helped |
| extend Gemma-2B with neutral residues rather than LoRA | Gemma-2B · French target learning | task average: 44.9 -> 48.2 | Helped |
| extend Gemma-2B with neutral residues rather than adapters | Gemma-2B · Hungarian target learning | task average: 38.4 -> 38.8 | Helped |
| extend Gemma-2B with neutral residues rather than adapters | Gemma-2B · Slovak target learning | task average: 37.7 -> 38.6 | Helped |
| extend Gemma-2B with neutral residues rather than finetuning | Gemma-2B · Math GSM8K (5-shot) | accuracy: 58.4 -> 54.4 | Hurt |
Surprising findings
- Regularizing adapter outputs with an l1 loss without a block gate does not work well and hurts learning of the new domain.
- Training on English Wikipedia only, used as a proxy for the original distribution, yields poor downstream performance, though mixing it at 10% helps.
Limitations
- Forgetting on English is still significant when training on math data, since Wikipedia proxy data does not disambiguate math English.
- Evaluation limited to language extension with perplexity and multiple-choice benchmarks; no generative or instruction-following evaluation.
Experimental setup
Datasets: CommonCrawl (Danish, French, Hungarian, Slovak, Japanese) English Wikipedia OpenMathInstruct-2 MetaMathQA FLORES
Benchmarks: ARC challenge HellaSwag MMLU CommonSenseQA Belebele PubMed perplexity GSM8K FLORES
Metrics: perplexity accuracy task average
Baselines: Finetuning LoRA Vanilla adapters Backbone LoRAMoE
Builds on: adapters LoRA mixture of experts GLU feed-forward
Closest papers by meaning
- Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning (NeurIPS 2025 · similarity 0.88)
- Boosting Multi-Domain Fine-Tuning of Large Language Models through Evolving Interactions between Samples (ICML 2025 · similarity 0.87)
- Beyond Next Token Prediction: Patch-Level Training for Large Language Models (ICLR 2025 · similarity 0.87)
- BBox-Adapter: Lightweight Adapting for Black-Box Large Language Models (ICML 2024 · similarity 0.87)
- Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning (ICLR 2024 · similarity 0.87)
- Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.