Research map / Architectures and efficiency / Pruning and efficient tuning
From PEFT to DEFT: Parameter Efficient Finetuning for Reducing Activation Density in Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DEFT, a density loss added to PEFT training (adapter, LoRA, prefix, prompt, QLoRA) that induces activation sparsity in MLP intermediate activations, plus ADA-DEFT with learnable layerwise gates that allow skipping MLP blocks at inference.
Problem
Fine-tuning large pretrained language models is costly, and dense MLP activations make inference expensive on hardware; the paper asks whether PEFT can also reduce activation density in transformers without hurting downstream performance.
Approach
Augment the PEFT task loss with a differentiable approximation of the L0 norm (tanh for ReLU models, l0 for GeLU models) of MLP intermediate activations, so that only the small set of PEFT parameters is trained to induce activation sparsity; ADA-DEFT adds per-block trainable scalars in [0,1] that can drive whole MLP blocks to zero and be skipped at inference.
Key results
- On GLUE with RoBERTa-Large, DEFT with Adapter reduces activation density by 44.94% on average relative to PEFT at comparable performance (88.06 vs 87.91 avg).
- On SQuAD with Flan-T5-XXL (11B) QLoRA, DEFT reduces encoder density by 53.19% and decoder density by 90.60% versus PEFT with F1 92.72 vs 92.84.
- ADA-DEFT reduces Flan-T5-XL inference runtime by 8.79% and memory by 17.46% relative to ADA-PEFT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add density loss to Adapter PEFT | RoBERTa Large · GLUE | average performance: 87.91 -> 88.06 | Helped |
| add density loss to PEFT | RoBERTa Large · GLUE | activation density: 44.94% density change | Helped |
| add density loss to Adapter PEFT | BERT Base · GLUE | average performance: 80.44 -> 80.67 | Helped |
| add density loss to Adapter PEFT | T5 Small · SQuAD | F1: $82.58\pm0.08$ -> $82.41\pm0.11$ | Hurt |
| add density loss to QLoRA PEFT | Flan-T5 XXL · SQuAD | encoder density: \textbf{53.19}% density change | Helped |
| add density loss to QLoRA PEFT | Flan-T5 XXL · SQuAD | decoder density: \textbf{90.60}% density change | Helped |
| add adaptive layerwise density weights | Flan-T5 XL QLoRA · SQuAD | F1: $93.11\pm0.04$ -> $92.30\pm0.00$ | Hurt |
| add adaptive layerwise density weights | Flan-T5 XL QLoRA · SQuAD | runtime: \textbf{8.79}% saving | Helped |
| apply WANDA pruning after DEFT instead of PEFT | RoBERTa Large · SST-2 and MNLI validation at 50% sparsity | accuracy: 51.73\% -> 60.96\% | Helped |
| replace tanh approximation with l0 approximation (ablation) | DEFT Adapter · SST-2 (BERT Base) | accuracy: {92.36 $\pm$ 0.36} -> 92.13 $\pm$ 0.30 | No clear effect |
| replace tanh approximation with sigmoid approximation (ablation) | DEFT Adapter · SST-2 (BERT Base) | accuracy: {92.36 $\pm$ 0.36} -> {92.02 $\pm$ 0.05} | No clear effect |
| replace tanh approximation with l1 approximation (ablation) | DEFT Adapter · SST-2 (BERT Base) | accuracy: {92.36 $\pm$ 0.36} -> {50.91 $\pm$ 0.00} | Hurt |
Surprising findings
- Prompt tuning on RoBERTa-Large with DEFT yields only ~1.44% density reduction and training collapses on CoLA.
- The l1 approximation of the density loss severely degrades BERT-Base performance (SST-2 accuracy 50.91 vs 92.36 for tanh).
Limitations
- Results across PEFT modules are not directly comparable due to differing parameter counts and placements.
- Energy savings are estimated via an ASIC simulator rather than measured hardware.
- Training can be unstable for some configurations (e.g., Prompt-T on CoLA, l1 approximation).
Experimental setup
Datasets: GLUE SQuAD CIFAR-10
Benchmarks: GLUE SQuAD CIFAR-10
Metrics: accuracy Matthews correlation Pearson correlation F1 exact match activation density density change energy consumption ratio runtime memory
Baselines: PEFT (Adapter) PEFT (LoRA) PEFT (Prefix-T) PEFT (Prompt-T) PEFT (QLoRA) ADA-PEFT
Builds on: LoRA Adapter Prefix-tuning Prompt tuning QLoRA WANDA
Closest papers by meaning
- Trainable Transformer in Transformer (ICML 2024 · similarity 0.88)
- S$^{2}$FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity (NeurIPS 2024 · similarity 0.87)
- WeGeFT: Weight‑Generative Fine‑Tuning for Multi‑Faceted Efficient Adaptation of Large Models (ICML 2025 · similarity 0.86)
- DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averaging (NeurIPS 2024 · similarity 0.86)
- Exploring the Benefit of Activation Sparsity in Pre-training (ICML 2024 · similarity 0.86)
- Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.