Research map / Large language models / Language, safety and interpretability
Beware of Calibration Data for Pruning Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies how calibration data affects post-training LLM pruning and proposes a self-generated calibration data strategy (prefix-truncated self-generation with perplexity filtering) that improves Wanda, DSnoT, and OWL.
Problem
Post-training pruning of LLMs relies on calibration data to estimate parameter importance, but the effect of calibration data source, amount, and similarity to pretraining data is under-explored, and pretraining data is often inaccessible.
Approach
Calibration data that is more similar to the pretraining data gives better pruned models, and since pretraining data is unavailable, the LLM itself can generate similar calibration data from short prefixes of candidate text, filtered by perplexity.
Key results
- On DCLM-7B at 60% sparsity, self-generated calibration data improved Wikipedia-calibrated Wanda by +2.59 average points and DSnoT by +2.68 points.
- Synthetic calibration data outperformed DCLM training-set calibration on LLaMA-2-7B and LLaMA-3-8B by about 0.3-0.7 average points.
- Similarity to pretraining data (Jaccard 3-gram MinHash: C4 0.070, SlimPajama 0.016, Wikipedia 0.008) tracked pruning performance better than data quality.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace original calibration data with self-generated synthetic data | Wanda on DCLM-7B · 60% sparsity, Wikipedia calibration | commonsense Avg. accuracy: 62.12 -> 64.71 (+2.59) | Helped |
| Replace original calibration data with self-generated synthetic data | DSnoT on DCLM-7B · 60% sparsity, Wikipedia calibration | commonsense Avg. accuracy: 61.09 -> 63.77 (+2.68) | Helped |
| Use self-generated synthetic calibration data | Wanda, DSnoT, and OWL on DCLM-7B · 60% sparsity, C4, SlimPajama, or DCLM calibration | commonsense Avg. accuracy: +0.37 to +1.43 | Helped |
| Use calibration data similar to pretraining data rather than Wikipedia | Wanda, DSnoT, and OWL on DCLM-7B · 2:4 semi-structured pruning | accuracy: DCLM: 62.88; Wikipedia: 61.03 (Wanda) | Helped |
| Increase calibration-data amount from 64 to 2048 samples | Wanda and DSnoT on DCLM-7B · 2:4 semi-structured pruning | average performance: 0.1%-0.2% | No clear effect |
| Use self-generated synthetic calibration data | Wanda on DCLM-7B · 50% sparsity | accuracy: 69.62 -> 69.64 | Helped |
| Use self-generated synthetic calibration data | Wanda on DCLM-7B · 65% sparsity | accuracy: 58.14 -> 58.11 | No clear effect |
| Use self-generated synthetic calibration data | Wanda on DCLM-7B · 4:8 semi-structured pruning | accuracy: 66.28 -> 67.02 | Helped |
| Use self-generated synthetic calibration data | Wanda on DCLM-7B · 2:4 semi-structured pruning | accuracy: 62.88 -> 63.61 | Helped |
| Filter the highest-perplexity synthetic samples at a 10%-20% rate (ablation) | Wanda on DCLM-7B · 60% sparsity, Wikipedia source | commonsense accuracy: 64.49 -> 64.76 (10% filter); 64.49 -> 64.71 (20% filter) | Helped |
| Increase perplexity filtering from 20% to 30%-40% (ablation) | Wanda on DCLM-7B · 60% sparsity, Wikipedia source | commonsense accuracy: 64.71 -> 64.49; 64.71 -> 64.51 | Hurt |
| Use a nonzero prefix when generating synthetic calibration data (ablation) | Wanda on DCLM-7B · Prefix-length ablation | commonsense accuracy | Helped |
Surprising findings
- Inappropriate calibration data (Wikipedia, SlimPajama) at 60% sparsity performed worse than magnitude pruning with no calibration data.
- Increasing calibration data amount did not narrow the performance gap between calibration sources.
Limitations
- Experiments limited to unstructured and semi-structured pruning on 7B-13B models; 70B and structured pruning not tested.
- Generation hyperparameters and filtering were not fully optimized.
- Does not address instruction-tuned or aligned LLMs as calibration targets.
Experimental setup
Datasets: C4 Wikipedia SlimPajama DCLM Alpaca
Benchmarks: BoolQ Winogrande PIQA Hellaswag ARC-e ARC-c MMLU Alpaca
Metrics: accuracy perplexity average accuracy
Baselines: Wanda DSnoT OWL magnitude pruning C4 calibration Wikipedia calibration SlimPajama calibration DCLM calibration
Builds on: Wanda SparseGPT DSnoT OWL Min-K%++
Closest papers by meaning
- Preserving LLM Capabilities through Calibration Data Curation: From Analysis to Optimization (NeurIPS 2025 · similarity 0.92)
- Calibrating Large Language Models with Sample Consistency (AAAI 2025 · similarity 0.91)
- Sample-aware Adaptive Structured Pruning for Large Language Models (AAAI 2025 · similarity 0.91)
- You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learning (ICLR 2025 · similarity 0.91)
- Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach (ICML 2025 · similarity 0.91)
- Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.