Research map / Architectures and efficiency / Pruning and efficient tuning
Increasing Model Capacity for Free: A Simple Strategy for Parameter Efficient Fine-tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CapaBoost, which applies static random binary masks to a shared weight matrix across parallel tied PEFT modules to raise the rank of incremental weights without adding parameters, improving LoRA and Adapter variants on GLUE, SQuAD, and VTAB-1k.
Problem
Parameter-efficient fine-tuning methods such as LoRA and adapters have capacity limited by the low inner rank of their incremental modules, especially under small parameter budgets.
Approach
Build several parallel weight-tied modules from one shared low-rank matrix by applying distinct static random sparsity masks to it, so the summed effective update has higher rank (about d times) at the same trainable parameter count; it plugs into LoRA, Adapter, and Prefix-Tuning.
Key results
- On GLUE with RoBERTa-base, CapaBoost-PAdapter reaches 86.31-86.35 average vs 84.30 for PAdapter, with 0.71% vs 0.95% trainable parameters.
- On SQuAD with DeBERTaV3-base, CapaBoost-PAdapter averages 86.5/90.9 EM/F1 across budgets, about 1.0/0.7 above the second-best LoRA.
- On VTAB-1k with ViT-B/16, CapaBoost-LoRA (d=4) averages 73.41 vs 72.38 for LoRA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply CapaBoost parallel masked modules to PAdapter | PAdapter on RoBERTa-base · GLUE | average score: 84.30 -> 86.31 | Helped |
| Apply CapaBoost parallel masked modules to LoRA | LoRA on RoBERTa-base · GLUE | average score: +1.9 | Helped |
| Apply CapaBoost to PAdapter with bottleneck dimension 32 | PAdapter on DeBERTaV3-base · GLUE | average score: 88.90 -> 89.68 | Helped |
| Apply CapaBoost to PAdapter | PAdapter on DeBERTaV3-base · SQuADv1.1 and SQuADv2.0 | average EM/F1: 85.5/90.2 -> 86.5/90.9 | Helped |
| Increase parallel tied modules from d=2 to d=4 (ablation) | CapaBoost-LoRA on ViT-B/16 · VTAB-1k | average score: 73.20 to 73.41 | Helped |
| Set mask density to 0.6 rather than other tested densities (ablation) | CapaBoost-LoRA on RoBERTa-base · CoLA | performance | Helped |
| Use distinct deterministic masks instead of same masks (ablation) | CapaBoost-PAdapter · GLUE | average score: 85.61 -> 86.33 | Helped |
| Use distinct deterministic masks instead of dropout masks (ablation) | CapaBoost-PAdapter · GLUE | average score: 85.06 -> 86.33 | Helped |
| Replace fully random masks with 2:4 sparsity random masks (ablation) | CapaBoost-LoRA on RoBERTa-base · GLUE | average score: 85.49 -> 85.51 | No clear effect |
| Increase LoRA inner dimension r in the single-layer CapaBoost variant (ablation) | CapaBoost-LoRA · CoLA | performance | Helped |
| Use two or three parallel layers rather than one (ablation) | CapaBoost-LoRA · CoLA | performance | Helped |
| Increase incremental-weight rank beyond the best-performing setting (ablation) | CapaBoost-LoRA · CoLA | performance | Hurt |
Surprising findings
- Dropout-style per-iteration masks perform poorly at low density, while static distinct masks help.
- On SQuADv2 with the smallest budget (0.08%), CapaBoost-PAdapter does not surpass full fine-tuning.
Limitations
- Overfitting when the inner dimension is set large, degrading performance.
- Uniform allocation across layers; adaptive allocation left for future work.
- Training FLOPs can be higher than a dense layer depending on sparsity and number of tied modules.
Experimental setup
Datasets: GLUE SQuADv1.1 SQuADv2.0 VTAB-1k
Benchmarks: GLUE SQuADv1.1 SQuADv2.0 VTAB-1k
Metrics: Matthews correlation accuracy Pearson/Spearman correlation EM/F1 rank
Baselines: Fine-tuning BitFit Prefix-Tuning HAdapter PAdapter LoRA AdaLoRA ProPETL VPT-Deep NOAH AdaMix Dropout
Builds on: LoRA Adapters Prefix-Tuning ProPETL AdaLoRA
Closest papers by meaning
- LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning (NeurIPS 2024 · similarity 0.87)
- VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks (NeurIPS 2024 · similarity 0.87)
- DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning (ICLR 2024 · similarity 0.87)
- Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.87)
- Gradient Weight-normalized Low-rank Projection for Efficient LLM Training (AAAI 2025 · similarity 0.87)
- MoLEx: Mixture of Layer Experts for Fine-tuning with Sparse Upcycling (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.