Research map / Architectures and efficiency / Pruning and efficient tuning
From Dense to Sparse: Contrastive Pruning for Better Pre-trained Language Model Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CAP contrastive supervision to movement pruning | Movement pruning · MNLI, QQP, and SQuAD across 50%, 90%, and 97% sparsity | average score: 1.0 (85.2→86.2), 1.3 (82.7→84.0), and 2.0 (77.8→79.8) | Helped |
| Add CAP contrastive supervision to first-order pruning | First-order pruning · MNLI, QQP, and SQuAD at 50% and 90% sparsity | average score: 1.9 (85.6→87.5) and 4.1 (78.7→82.8) | Helped |
| Add PrC, contrastive learning from the pre-trained model (ablation) | CAP-first-order · MNLI-m, QQP, and SQuAD at 90% sparsity | average score: -1.78, -0.87, -5.11 | Helped |
| Add SnC, contrastive learning from pruning snapshots (ablation) | CAP-first-order · MNLI-m, QQP, and SQuAD at 90% sparsity | average score: -1.57, -1.72, -4.11 | Helped |
| Add FiC, contrastive learning from the fine-tuned model (ablation) | CAP-first-order · MNLI-m, QQP, and SQuAD at 90% sparsity | average score: -1.53, -1.31, -2.64 | Helped |
| Combine supervised and unsupervised contrastive objectives (ablation) | CAP · MNLI and QQP at 50% and 90% sparsity | average score: 0.53~0.91; 0.49~1.25 | Helped |
| Add knowledge distillation (ablation) | CAP · SQuAD, token-level task | F1: 88.73 -> 86.52; 80.59 -> 77.76 | Helped |
| Add knowledge distillation (ablation) | CAP · QQP, sentence-level task | F1: 88.58 -> 88.59; 86.92 -> 86.88; 88.62 -> 88.60; 87.65 -> 87.67 | No clear effect |
| Use [CLS] pooling instead of mean pooling (ablation) | CAP · QQP | performance | Helped |
Closest papers by meaning
- SparseLLM: Towards Global Pruning of Pre-trained Language Models (NeurIPS 2024 · similarity 0.93)
- Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision (ICML 2025 · similarity 0.91)
- The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws (ICLR 2025 · similarity 0.91)
- Compact Language Models via Pruning and Knowledge Distillation (NeurIPS 2024 · similarity 0.91)
- Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation (NeurIPS 2025 · similarity 0.91)
- Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.