Research map / Architectures and efficiency / Pruning and efficient tuning
Parameter-Efficient Transfer Learning for NLP
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use bottleneck adapters instead of full fine-tuning | BERT_LARGE · GLUE test | Total score: 80.4 -> 80.0 | No clear effect |
| Use fixed adapter size 64 instead of selecting size per task (ablation) | BERT_LARGE adapters · GLUE test | Total score: 80.0 -> 79.6 | Hurt |
| Use bottleneck adapters instead of full fine-tuning | BERT_BASE · Additional classification tasks test | Average accuracy: 73.7 -> 73.3 | No clear effect |
| Use variable top-layer fine-tuning instead of full fine-tuning | BERT_BASE · Additional classification tasks test | Average accuracy: 73.7 -> 74.0 | Helped |
| Fine-tune only the top BERT_BASE layer rather than use adapters of size 64 | BERT_BASE · MNLI_m validation | Accuracy: 77.8 % ± 0.1 % -> 83.7% ± 0.1 % | Hurt |
| Use adapters of size 64 instead of full fine-tuning | BERT · SQuAD v1.1 validation | F1: 90.7 -> 90.4% | No clear effect |
| Reduce adapter size to 2 (ablation) | BERT adapters · SQuAD v1.1 validation | F1: 90.4% -> 89.9 | Hurt |
| Remove all trained adapters (ablation) | BERT_BASE adapters · MNLI validation | Accuracy: drops substantially: to 37% | Hurt |
| Remove all trained adapters (ablation) | BERT_BASE adapters · CoLA validation | Accuracy: drops substantially: to 69% | Hurt |
| Use adapter initialization standard deviation that is too large (ablation) | BERT adapters · MNLI and CoLA | Performance | Hurt |
| Use a fixed adapter size across tasks rather than selecting task-specific sizes (ablation) | BERT adapters · Eight classification tasks | Mean validation accuracy: 86.2% -> 85.8% -> 85.7% | No clear effect |
| Add batch/layer normalization, extra adapter layers, alternative activations, attention-only placement, or parallel adapters (ablation) | Bottleneck adapter architecture · Multiple datasets | Performance | No clear effect |
Closest papers by meaning
- Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass (ICLR 2025 · similarity 0.86)
- Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models (ICML 2025 · similarity 0.86)
- TAIA: Large Language Models are Out-of-Distribution Data Learners (NeurIPS 2024 · similarity 0.86)
- On the Duality between Gradient Transformations and Adapters (ICML 2025 · similarity 0.86)
- FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain (ICML 2025 · similarity 0.86)
- Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspective (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.