Research map / Architectures and efficiency / Pruning and efficient tuning
Fine-Tuning Pre-Trained Language Models Effectively by Optimizing Subnetworks Adaptively
Zhang Haojie, Ge Li, Jia Li, Zhongjin Zhang, YUQI ZHU, Zhi Jin
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace static CHILD-TUNING_D subnetwork with dynamic gradient-based selection (DPS Dense) | BERT-large fine-tuning · GLUE CoLA/MRPC/RTE/STS-B | average score: 79.62 -> 80.03 | Improved |
| replace Mixout random subnetwork with frequency-penalized dynamic selection (DPS Mix) | BERT-large fine-tuning · GLUE CoLA/MRPC/RTE/STS-B | average score: 79.54 -> 80.21 | Improved |
| use DPS Mix instead of vanilla fine-tuning | BERT-large · GLUE 4 tasks | average std: 1.64 -> 0.84 | Improved |
| use DPS Mix instead of vanilla fine-tuning | BERT-large trained on MNLI 8K · out-of-domain NLI average | accuracy: 72.01 -> 73.87 | Improved |
| use DPS Mix instead of CHILD-TUNING_D | BERT-large trained on SNLI 8K · out-of-domain NLI average | accuracy: 69.79 -> 70.41 | Improved |
| use DPS instead of vanilla fine-tuning with 0.5K training examples | BERT-large · GLUE low-resource avg | average score: 65.85 -> 69.12 | Improved |
| use DPS instead of vanilla fine-tuning with 1K training examples | BERT-large · GLUE low-resource avg | average score: 73.19 -> 75.04 | Improved |
| use DPS Mix on other pre-trained models (RoBERTa, ELECTRA, BART, DeBERTa) | DPS · GLUE 4 tasks avg | average score: 84.66 -> 86.67 (ELECTRA) | Improved |
| vary subnetwork probability p from 0.1 to 0.5 (ablation) | DPS vs CHILD-TUNING_D and Mixout · CoLA, RTE | validation score | Improved |
| use DPS on large training sets (SST-2, QNLI) | BERT-large · SST-2 dev | accuracy: 93.23 -> 93.77 | Improved |
| use DPS Dense instead of CHILD-TUNING_D in time | BERT-large on RTE · fine-tuning time | relative time: x3.13 -> x1.12 | Improved |
| use DPS Mix instead of vanilla in memory (batch 16) | BERT-large · memory consumption | GB: 10.6G -> 21.1G | Hurt |
Closest papers by meaning
- Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMs (ICLR 2024 · similarity 0.90)
- Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models (ICML 2025 · similarity 0.90)
- Enhancing Large Language Model Performance with Gradient-Based Parameter Selection (AAAI 2025 · similarity 0.90)
- Selecting Large Language Model to Fine-tune via Rectified Scaling Law (ICML 2024 · similarity 0.90)
- Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning (NeurIPS 2025 · similarity 0.89)
- Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
