Research map / Architectures and efficiency / Pruning and efficient tuning
DARE the Extreme: Revisiting Delta-Parameter Pruning For Fine-Tuned Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Revisits delta-parameter pruning (DARE) and proposes DAREx-q (tuned rescaling factor 1/q instead of 1/(1-p)) and DAREx-L2 (AdamR regularized fine-tuning) to enable 99%+ delta pruning, plus importance-based DPP analysis.
Problem
Random drop-and-rescale (DARE) delta-parameter pruning degrades at high pruning rates or when delta parameters are large, limiting storage savings for fine-tuned models.
Approach
Bound absolute output changes from DARE to show the rescaling factor 1/(1-p) and the mean/variance of delta-induced influences drive failure; then replace the rescaling factor with a tuned 1/q (global or per-layer, chosen on validation data or unlabeled output change) and fine-tune with AdamR (L2 or L1 regularization on the delta parameters) to make deltas highly prunable.
Key results
- On BERT with p=0.99, DARE gets 4.25 on CoLA and 51.00 on SST-2, while DAREx-L2 reaches 57.24 and 88.17 (no-pruning 56.24 and 90.25).
- DAREx-q with per-layer 1/q_v improves over DARE by over 40 points on CoLA and SST-2 at p=0.99 for BERT (e.g., 53.12 vs 4.25 on CoLA).
- On GSM8K with p=0.99, vanilla DARE scores 0.00 for Abel-7B, MetaMath-7B and WizardMath-7B, while DAREx-q (1/q_v) reaches 20.20, 34.87 and 35.63 respectively.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace DARE rescaling with tuned 1/q rescaling | BERT delta pruning · CoLA (p=0.99) | Matthews correlation coefficient (%): 4.25 -> 48.96 (1/q_v); 4.25 -> 45.20 (1/q_e) | Helped |
| Apply DARE after AdamR-L2 fine-tuning | BERT delta pruning · CoLA (p=0.99) | Matthews correlation coefficient (%): 4.25 -> 57.24 (+52.99) | Helped |
| Use per-layer instead of global rescaling (ablation) | DAREx-q · Encoder datasets at p=0.99 | test performance | Helped |
| Use global instead of per-layer rescaling (ablation) | DAREx-q · Decoder GSM8K at p=0.95 and p=0.99 | zero-shot accuracy | No clear effect |
| Tune rescaling with unlabeled data rather than labeled validation data (ablation) | DAREx-q · Encoder datasets at p=0.99 | test performance | No clear effect |
| Apply AdamR-L2 regularization before DARE | BERT delta pruning · SST-2 (p=0.99) | accuracy (%): 51.00 -> 88.17 (+37.17) | Helped |
| Apply AdamR-L2 regularization before DARE | RoBERTa delta pruning · STS-B (p=0.99) | mean Pearson and Spearman correlation: 81.59 -> 89.88 (+8.29) | Helped |
| Apply AdamR-L1 regularization before magnitude pruning (ablation) | BERT delta pruning · CoLA (p=0.99) | Matthews correlation coefficient (%): 8.49 -> 12.30 | Helped |
| Apply AdamR-L1 regularization before WANDA pruning (ablation) | WANDA on encoder delta parameters · SST-2 and STS-B | test performance: 86.93 -> 79.93; 76.15 -> 72.82 | Mixed |
| Use importance-based pruning rather than DARE for large deltas | MetaMath-Llemma-7B delta parameters · GSM8K (p=0.5) | zero-shot accuracy: 0.00 -> 47.50 (MP); 0.00 -> 54.50 (WANDA) | Helped |
| Prune LoRA parameters with tuned DAREx-q instead of vanilla DARE | BERT-Base LoRA · SST-2 (p=0.9) | accuracy (%): 48.05 -> 90.25 | Helped |
Surprising findings
- Vanilla DARE fails completely (0.00 GSM8K) at 99% pruning on several 7B math models, while simple magnitude pruning and WANDA retain up to p=0.5 with minimal loss on MetaMath-llema-7B.
- WANDA with AdamR-L1 hurt performance on some datasets (e.g., SST-2 and STS-B) because pruning ignores small-activation dimensions.
Limitations
- Decoder experiments are limited to a 0.5B Qwen2 model fine-tuned by the authors plus public 7B models; scale coverage is narrow.
- The rescaling-factor search requires validation labels or a batch of unlabeled inference, and per-layer tuning adds search cost.
- WANDA does not consistently benefit from AdamR-L1 regularization.
Experimental setup
Datasets: CoLA SST-2 MRPC STS-B MetaMath GSM8K CIFAR-10 SVHN
Benchmarks: CoLA SST-2 MRPC STS-B GSM8K
Metrics: Matthews correlation accuracy accuracy and F1 average Pearson/Spearman mean zero-shot accuracy
Baselines: DARE magnitude pruning WANDA Random Drop
Builds on: DARE AdamW WANDA magnitude pruning LoRA
Closest papers by meaning
- Sparsest Models Elude Pruning: An Exposé of Pruning’s Current Capabilities (ICML 2024 · similarity 0.88)
- Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision (ICML 2025 · similarity 0.88)
- Sparse Model Soups: A Recipe for Improved Pruning via Model Averaging (ICLR 2024 · similarity 0.88)
- POP: Online Structural Pruning Enables Efficient Inference of Large Foundation Models (NeurIPS 2026 · similarity 0.87)
- Discovering Sparsity Allocation for Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.87)
- The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.