Research map / Large language models / Instruction and fine-tuning
Cherry on Top: Parameter Heterogeneity and Quantization in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reveals parameter heterogeneity in LLMs (a ~1% subset of "cherry" parameters has outsized loss impact) and proposes CherryQ, a QAT-based mixed-precision quantizer that keeps cherry parameters in FP16 while quantizing the rest end-to-end.
Problem
Post-training quantization of LLMs to low bit-widths degrades performance, and existing mixed-precision methods struggle to identify important parameters and to jointly optimize high-precision and low-precision parameters.
Approach
Quantify parameter impact via the diagonal Hessian (approximated by the Fisher information), show that a tiny fraction of parameters dominate it, keep those "cherry" parameters in FP16, and train the whole mixed-precision model end-to-end with gradient descent for cherry parameters and STE for normal parameters.
Key results
- On LLaMA2-7B 3-bit g128, CherryQ reaches C4 perplexity 7.39 and WikiText-2 5.93 vs. 7.75/6.03 for OmniQuant (FP16: 6.97/5.47).
- 3-bit Vicuna-1.5 with CherryQ achieves a win-tie-lose ratio against its FP16 counterpart slightly better than even according to GPT-4 judging on Vicuna-bench.
- On 2-bit LLaMA2-7B g128, CherryQ reaches C4 perplexity 9.55 vs. 15.02 for OmniQuant and 33.70 for GPTQ.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CherryQ instead of OmniQuant for 3-bit quantization | LLaMA2-7B-3bit-g128 · C4 validation | perplexity: 7.75 -> 7.39 | Helped |
| Use CherryQ instead of OmniQuant for 3-bit quantization | LLaMA2-7B-3bit-g128 · WikiText-2 | perplexity: 6.03 -> 5.93 | Helped |
| Use CherryQ instead of SqueezeLLM for 3-bit quantization | LLaMA2-7B-3bit-g64 · C4 validation | perplexity: 7.51 -> 7.34 | Helped |
| Use CherryQ instead of SqueezeLLM for 3-bit quantization | LLaMA2-13B-3bit-g128 · WikiText-2 | perplexity: 5.23 -> 5.26 | Hurt |
| Use CherryQ instead of OmniQuant for 4-bit quantization | LLaMA2-13B-4bit-g128 · WikiText-2 | perplexity: 4.95 -> 4.99 | Hurt |
| Use CherryQ instead of OmniQuant for 2-bit quantization | LLaMA2-7B-2bit-g128 · C4 validation | perplexity: 15.02 -> 9.55 | Helped |
| Use CherryQ instead of QAT | LLaMA2-7B-3bit-g64 · OpenLLM average | average score: 46.9 -> 48.7 | Helped |
| Use CherryQ instead of QAT | Vicuna-1.5-13B-3bit-g128 · zero-shot MMLU | accuracy: 50.9 -> 51.9 | Helped |
| Select cherry parameters by impact rather than weight (ablation) | LLaMA2-7B-3bit-g64 · C4 | perplexity: 7.93 -> 7.34 | Helped |
| Select cherry parameters by impact rather than activation (ablation) | LLaMA2-7B-3bit-g64 · C4 | perplexity: 7.37 -> 7.34 | Helped |
| Select cherry parameters by impact rather than activation (ablation) | LLaMA2-7B-3bit-g64 · WikiText-2 | perplexity: 5.89 -> 5.87 | Helped |
Surprising findings
- 3-bit CherryQ Vicuna-1.5 was judged slightly better than its FP16 counterpart by GPT-4, which the authors attribute to GPT-4 difficulty distinguishing outputs rather than true superiority.
Limitations
- Performance depends on accurate identification of cherry parameters, which may vary across architectures and datasets.
- Computational overhead of impact-based identification and scaling may be challenging for very large or latency-sensitive deployments.
- No error bars reported due to computational cost.
Experimental setup
Datasets: C4 WikiText-2 ShareGPT Vicuna-bench MMLU HuggingFace OpenLLM Leaderboard tasks
Benchmarks: C4 WikiText-2 HuggingFace OpenLLM Leaderboard Vicuna-bench MMLU
Metrics: perplexity accuracy win-tie-lose ratio
Baselines: QAT GPTQ SqueezeLLM OmniQuant AWQ
Builds on: GPTQ OmniQuant SqueezeLLM AWQ LLM-QAT
Closest papers by meaning
- What Makes Quantization for Large Language Model Hard? An Empirical Study from the Lens of Perturbation (AAAI 2024 · similarity 0.91)
- QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models (ICLR 2024 · similarity 0.91)
- SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models (ICML 2025 · similarity 0.91)
- Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs (ICML 2024 · similarity 0.91)
- Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantization (NeurIPS 2025 · similarity 0.90)
- Evaluating Quantized Large Language Models (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.