Research map / Large language models / Language, safety and interpretability
Treasures in Discarded Weights for LLM Quantization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Discarded Weight Recycling (DWR), a plug-and-play post-quantization step that builds low-rank search spaces around the discarded quantization residual and merges candidates selected by calibration-set perplexity, improving low-bit LLM quantization without extra inference cost.
Problem
Low-bit (INT4/INT3/INT2) weight quantization of LLMs causes significant accuracy loss; existing PTQ methods adjust weights layer-by-layer and ignore global information, and the residual discarded by quantization is unused.
Approach
Treat the discarded weights D = W - Wq as a resource: generate low-rank search spaces around D (random orthogonal, SVD, or AFM/PCA-based) and merge into the quantized weights the candidate that minimizes global calibration-set perplexity, keeping the quantization grid so inference cost is unchanged.
Key results
- On LLaMA2-70B with GPTQ INT3, DWR raises average commonsense accuracy from 67.49 to 67.85 and reduces WikiText2 perplexity from 3.88 to 3.40.
- On LLaMA3-8B with GPTQ INT3, DWR lowers WikiText2 perplexity from 67.06 to 6.34 and raises average accuracy from 55.62 to 66.00.
- On LLaMA2-7B with OmniQuant INT2, DWR lowers WikiText2 perplexity from 11.23 to 10.41 and raises average accuracy from 46.35 to 47.06.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DWR after GPTQ INT4 | BLOOM-7B1 · zero-shot commonsense QA | Avg. accuracy: 52.32 -> 52.96 | Helped |
| Add DWR after GPTQ INT3 | LLaMA2-7B · MMLU 0-shot | Avg. accuracy: 36.44 -> 41.63 | Helped |
| Add DWR before QA-LoRA fine-tuning | LLaMA2-7B QA-LoRA INT4 · MMLU 0-shot | Avg. accuracy: 43.60 -> 45.64 | Helped |
| Use random search-space generation (ablation) | BLOOM-7B1 GPTQ INT4 + DWR · zero-shot commonsense QA | ACC.: 52.32 -> 52.49 | Helped |
| Use SVD search-space generation (ablation) | BLOOM-7B1 GPTQ INT4 + DWR · zero-shot commonsense QA | ACC.: 52.32 -> 52.65 | Helped |
| Replace PPL criterion with direct merging (RTN) (ablation) | BLOOM-7B1 GPTQ INT3 + DWR · WikiText2 | perplexity: 11.97 -> 12.53 | Hurt |
| Replace PPL criterion with KL criterion (ablation) | BLOOM-7B1 GPTQ INT3 + DWR · WikiText2 | perplexity: 11.97 -> 12.25 | Hurt |
| Vary the calibration sample count, skipped blocks, or search interval (ablation) | BLOOM-7B1 GPTQ INT4 + DWR · WikiText2 | perplexity | No clear effect |
Surprising findings
- GPTQ INT3 on LLaMA3-8B collapses (WikiText2 perplexity 67.06), which DWR almost fully recovers to 6.34.
- Even a random orthogonal search space improves over the base GPTQ model, suggesting the perplexity-guided search rather than the specific space drives gains.
Limitations
- Only weight-only quantization is studied; activation quantization is not handled.
- Evaluated only on dense LLaMA2/LLaMA3/BLOOM models, not multimodal or MoE models.
- Requires a calibration set and per-layer perplexity search, adding offline compute time.
Experimental setup
Datasets: C4 WikiText2 Flan v2
Benchmarks: WikiText2 C4 BoolQ PIQA SIQA HellaSwag WinoGrande ARC-easy ARC-challenge OpenBookQA MMLU
Metrics: perplexity accuracy
Baselines: GPTQ OmniQuant QA-LoRA RTN merging KL-based criterion LoWC SVD generation random generation
Builds on: GPTQ OmniQuant QA-LoRA AFM (PCA-based compression) AWQ
Closest papers by meaning
- BiLLM: Pushing the Limit of Post-Training Quantization for LLMs (ICML 2024 · similarity 0.92)
- What Makes Quantization for Large Language Model Hard? An Empirical Study from the Lens of Perturbation (AAAI 2024 · similarity 0.91)
- Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs (ICML 2024 · similarity 0.91)
- PB-LLM: Partially Binarized Large Language Models (ICLR 2024 · similarity 0.91)
- Norm Tweaking: High-Performance Low-Bit Quantization of Large Language Models (AAAI 2024 · similarity 0.91)
- ARB-LLM: Alternating Refined Binarizations for Large Language Models (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.