AI Paper Atlas

Research map / Architectures and efficiency / Pruning and efficient tuning

Fine-Tuning Pre-Trained Language Models Effectively by Optimizing Subnetworks Adaptively

Zhang Haojie, Ge Li, Jia Li, Zhongjin Zhang, YUQI ZHU, Zhi Jin

NeurIPS 2022 · Architectures and efficiency › Pruning and efficient tuning · approach: Adapter-based parameter-efficient tuning · 12 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace static CHILD-TUNING_D subnetwork with dynamic gradient-based selection (DPS Dense)BERT-large fine-tuning · GLUE CoLA/MRPC/RTE/STS-Baverage score: 79.62 -> 80.03Improved
replace Mixout random subnetwork with frequency-penalized dynamic selection (DPS Mix)BERT-large fine-tuning · GLUE CoLA/MRPC/RTE/STS-Baverage score: 79.54 -> 80.21Improved
use DPS Mix instead of vanilla fine-tuningBERT-large · GLUE 4 tasksaverage std: 1.64 -> 0.84Improved
use DPS Mix instead of vanilla fine-tuningBERT-large trained on MNLI 8K · out-of-domain NLI averageaccuracy: 72.01 -> 73.87Improved
use DPS Mix instead of CHILD-TUNING_DBERT-large trained on SNLI 8K · out-of-domain NLI averageaccuracy: 69.79 -> 70.41Improved
use DPS instead of vanilla fine-tuning with 0.5K training examplesBERT-large · GLUE low-resource avgaverage score: 65.85 -> 69.12Improved
use DPS instead of vanilla fine-tuning with 1K training examplesBERT-large · GLUE low-resource avgaverage score: 73.19 -> 75.04Improved
use DPS Mix on other pre-trained models (RoBERTa, ELECTRA, BART, DeBERTa)DPS · GLUE 4 tasks avgaverage score: 84.66 -> 86.67 (ELECTRA)Improved
vary subnetwork probability p from 0.1 to 0.5 (ablation)DPS vs CHILD-TUNING_D and Mixout · CoLA, RTEvalidation scoreImproved
use DPS on large training sets (SST-2, QNLI)BERT-large · SST-2 devaccuracy: 93.23 -> 93.77Improved
use DPS Dense instead of CHILD-TUNING_D in timeBERT-large on RTE · fine-tuning timerelative time: x3.13 -> x1.12Improved
use DPS Mix instead of vanilla in memory (batch 16)BERT-large · memory consumptionGB: 10.6G -> 21.1GHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy