Research map / Large language models / Language, safety and interpretability
Preserving LLM Capabilities through Calibration Data Curation: From Analysis to Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematic study of how calibration data composition and domain correspondence affect LLM capability preservation under pruning and quantization, plus COLA, a three-stage activation-space calibration data curation framework.
Problem
Calibration data for post-training LLM compression (pruning and quantization) is usually assumed robust, and prior studies only examine perplexity or commonsense reasoning from isolated angles; the effect on high-level reasoning (math, code) and the mechanism behind calibration quality remain unclea
Approach
Analyzes calibration data along compositional properties (sequence length, sample count, source, format, language) and domain correspondence (subject, difficulty), attributes quality to representativeness and diversity in activation space, and curates calibration sets by randomly projecting layer activations, k-means clustering, and selecting samples nearest cluster centroids.
Key results
- Under general deployment, COLA improves AWQ 4-bit LLaMA3-8B math from 36.46 to 37.85 and code from 38.71 to 40.17 relative to WikiText random calibration.
- For targeted AWQ deployment, COLA raises Qwen2.5-7B code from 68.73% (CodeQA random) to 72.18%, a 3.45-point gain.
- Calibration on MathQA raises math performance by 5.92 points for quantization methods, per the paper's subject analysis.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use COLA activation-aware curated calibration data instead of random or Self-Gen calibration | Compressed LLMs in general deployment · General deployment: commonsense, math, and code | Accuracy: 44.23% vs. 41.85% for SparseGPT on LLaMA3-8B commonsense | Helped |
| Use COLA instead of random domain-specific calibration | AWQ (4-bit), LLaMA3-8B · Targeted math deployment | Math accuracy: 41.85% to 44.35% | Helped |
| Use COLA instead of random domain-specific calibration | AWQ (4-bit), Qwen2.5-7B · Targeted code deployment | Code accuracy: 68.73% to 72.18% | Helped |
| Use question-answer calibration with explicit reasoning chains instead of question-only format (ablation) | GPTQ (4-bit), LLaMA3-8B · Commonsense reasoning | Accuracy: 68.27% vs. 63.21% | Helped |
| Use Q&A with explicit reasoning chains instead of standard pre-training calibration (ablation) | AWQ (4-bit), Qwen2.5-7B · Math | Accuracy: 51.84% compared to 47.34% | Helped |
| Use English rather than Japanese calibration (ablation) | GPTQ (4-bit), LLaMA3-8B · Math | Accuracy: 31.22% vs. 16.72% | Helped |
| Use subject-aligned MathQA calibration rather than WikiText (ablation) | Compressed LLMs · Math | Accuracy: +5.92 percentage points for quantization methods and +4.35 points for pruning methods | Helped |
| Use subject-aligned CodeQA calibration rather than WikiText (ablation) | Compressed LLMs · Code generation | Accuracy: +7.49 points for quantization and +4.28 points for pruning | Helped |
| Use subject-specific calibration data rather than WikiText (ablation) | Compressed LLMs · Language modeling | Perplexity: average increase of 2.73 points for quantization and 3.52 points for pruning | Hurt |
| Use longer calibration sequences (ablation) | Pruning methods · Math problem solving | Accuracy: 25.5% drop at short sequences | Helped |
| Increase calibration samples from 16 to 128 (ablation) | AWQ, LLaMA3-8B · Code generation | Accuracy: 46.40% to 38.71% | Hurt |
| Increase calibration samples from 16 to 128 (ablation) | GPTQ, Qwen2.5-7B · Code generation | Accuracy: 57.67% to 34.03% | Hurt |
Surprising findings
- Calibration data source can change capability preservation by more than the choice of compression method itself.
- Pruning methods are reported to be more sensitive to sample count and sequence length than quantization methods, with AWQ largely robust to small sample sizes.
Limitations
- Improvements over baselines are modest (roughly 1-2 points in the general setting).
- Only two models (LLaMA3-8B-Instruct and Qwen2.5-7B-Instruct) and four compression methods are studied; structured pruning is excluded.
- The spectral mechanism analysis is largely qualitative and the link between activation diversity and capability preservation is asserted more than rigorously shown.
Experimental setup
Datasets: C4 WikiText SlimPajama CommonsenseQA MathQA CodeQA Easy2Hard-Bench
Benchmarks: WikiText2 PTB BoolQ PIQA HellaSwag WinoGrande ARC-Easy ARC-Challenge OpenbookQA GSM8K MATH Minerva-Math
Metrics: perplexity accuracy pass@1
Baselines: WikiText random calibration C4 random calibration SlimPajama random calibration Self-Generating then Sampling (Self-Gen)
Builds on: SparseGPT Wanda GPTQ AWQ
Closest papers by meaning
- Beware of Calibration Data for Pruning Large Language Models (ICLR 2025 · similarity 0.92)
- Calibrating Large Language Models with Sample Consistency (AAAI 2025 · similarity 0.91)
- Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.90)
- Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression (ICML 2025 · similarity 0.90)
- Compressing LLMs: The Truth is Rarely Pure and Never Simple (ICLR 2024 · similarity 0.89)
- Understanding LLM Behaviors via Compression: Data Generation, Knowledge Acquisition and Scaling Laws (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.