Research map / Architectures and efficiency / Pruning and efficient tuning
Junk DNA Hypothesis: Pruning Small Pre-Trained Weights $\textit{Irreversibly}$ and $\textit{Monotonically}$ Impairs ``Difficult" Downstream Tasks in LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the Junk DNA Hypothesis: magnitude-pruning small pre-trained LLM weights monotonically and irreversibly impairs harder downstream tasks, unlike quantization.
Problem
Whether small-magnitude pre-trained weights in LLMs are truly redundant, and how their removal affects downstream tasks of varying difficulty.
Approach
Defines task difficulty along controllable axes (data adequacy, option count, context length, k-shot, human-LLM gap, factoid vs MCQ) and shows that magnitude pruning degrades harder tasks monotonically (quantized models do not), with Theil-Sen slopes and Spearman correlation quantifying the trend; fine-tuning cannot recover the lost knowledge.
Key results
- Pruning shows monotonic impairment across task difficulty: Spearman correlations of 0.975-1.0 for pruning versus 0.316-0.872 for GPTQ quantization across context-length and option-count settings (Table 2).
- On the easiest ICL setting (5 shots on MMLU), up to 14% of small-magnitude weights can be removed with no drop, while the hardest setting (0 shots) is already impaired.
- Fine-tuning cannot recover knowledge from pruned small weights: Sparse-to-Dense transfer still underperforms Dense Transfer, and freezing up to 70% of small weights costs nothing on harder tasks only when moderate updates are allowed.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Prune increasing fractions of small-magnitude pre-trained weights | Vicuna-7B · TriviaQA and MMLU difficulty spectra | Downstream performance degradation across task difficulty: 0.996 vs 0.799; 0.979 vs 0.872 | Mixed |
| Replace pruning with GPTQ quantization | Vicuna-7B · TriviaQA and MMLU difficulty spectra | Spearman rank correlation: 0.996 vs 0.799; 0.979 vs 0.872 | Hurt |
| Use uniform SVD low-rank compression instead of pruning | Vicuna-7B · TriviaQA and MMLU difficulty spectra | Spearman rank correlation: 0.899; 0.942; 0.905; 0.912; 0.933; 0.956; 0.956 | Helped |
| Prune small-magnitude weights before downstream fine-tuning | Sparse Transfer · Difficult downstream tasks | Task performance | Hurt |
| Allow pruned weights to regain non-zero values during fine-tuning (ablation) | Sparse to Dense Transfer · Difficult downstream tasks | Task performance | Hurt |
| Freeze pre-trained small-magnitude weights during fine-tuning (ablation) | Dense Transfer · SST-2, MNLI, and QNLI | Task performance: 90% | No clear effect |
| Freeze pre-trained small-magnitude weights during fine-tuning (ablation) | Dense Transfer · COLA, CSQA, and WinoGrande | Task performance: 70% | No clear effect |
| Use SparseGPT or Wanda weight-importance pruning (ablation) | Vicuna-7B · Downstream tasks across difficulty levels | Performance gap between easy and challenging tasks | Mixed |
| Use structured N:M pruning instead of unstructured pruning (ablation) | RoBERTa-Base and Vicuna models · Downstream tasks across difficulty levels | Task performance degradation with difficulty | No clear effect |
| Increase the amount of magnitude pruning | Pre-trained LLMs · Downstream tasks across difficulty levels | Task performance | Hurt |
| Prune up to 14% of small weights | Vicuna-7B · MMLU with 5 ICL demonstrations | Accuracy: 14% | No clear effect |
| Prune around 10-15% of small weights | Models in the four within-task settings · Easier downstream tasks | Task performance: 10-15% | No clear effect |
Surprising findings
- Quantization does not show the monotonic impairment pattern that pruning does, and can even slightly benefit overall performance.
- SparseGPT and Wanda tend to benefit simple tasks more than challenging tasks.
Limitations
- Task difficulty is defined by proxy settings (data volume, option count, context length, human-LLM gap), and the across-task human gap metric is acknowledged as unreliable when both humans and models perform poorly.
- Focuses on one-shot pruning and uniform SVD/GPTQ settings rather than comprehensive compression methods; does not re-justify LLM compressibility.
Experimental setup
Datasets: GLUE MMLU TriviaQA PIQA HellaSwag OpenBookQA FreebaseQA CSQA WinoGrande
Benchmarks: GLUE MMLU TriviaQA PIQA HellaSwag OpenBookQA FreebaseQA CSQA WinoGrande
Metrics: normalized performance drop Spearman rank correlation Theil-Sen slope accuracy
Baselines: SparseGPT Wanda GPTQ Magnitude pruning Sparse Transfer
Builds on: Lottery Ticket Hypothesis SparseGPT Wanda
Closest papers by meaning
- Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment (AAAI 2025 · similarity 0.88)
- The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs (ICLR 2026 · similarity 0.88)
- A Simple and Effective Pruning Approach for Large Language Models (ICLR 2024 · similarity 0.87)
- Training on the Test Task Confounds Evaluation and Emergence (ICLR 2025 · similarity 0.87)
- Predicting Emergent Abilities with Infinite Resolution Evaluation (ICLR 2024 · similarity 0.87)
- Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.