Research map / Architectures and efficiency / Quantization
ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Post-training conversion of pretrained LLMs into multiplication-free models by quantizing weights into binary-coding (BCQ) matrices with power-of-two scaling factors, plus multi-objective error optimization and automated mixed bit allocation.
Problem
Deploying LLMs on resource-constrained hardware is limited by parameter size and dense multiplications; existing shift-and-add reparameterizations need training from scratch or full fine-tuning, which is too costly for LLMs.
Approach
Reparameterize each pretrained weight matrix as a sum of binary matrices with group-wise scaling factors (BCQ), quantize scaling factors to powers of two so multiplications become shifts, and replace remaining multiplications with LUT queries and adds; optimize a combined weight-plus-activation error objective and allocate bits per layer via an integer program driven by a sensitivity criterion.
Key results
- At 3 bits, Ours (Acc.) reduces average WikiText-2 perplexity by 5.63 vs OPTQ, 38.47 vs LUT-GEMM and 5136.13 vs AWQ on OPT models.
- At 2 bits, Ours (Acc.) achieves an average perplexity reduction of 22.74 over the strongest baseline QuIP on OPT models.
- Memory for OPT-66B drops from 122GB (FP16) to 23GB at 3 bits and 16GB at 2 bits; estimated Eyeriss energy falls about 87% for OPT-66B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply 3-bit ShiftAddLLM instead of OPTQ | OPT-66B · WikiText-2 | perplexity: 14.16 -> 9.43 | Helped |
| Apply 3-bit ShiftAddLLM instead of AWQ | LLaMA-2-70B · WikiText-2 | perplexity: 3.74 -> 3.64 | Helped |
| Apply ShiftAddLLM instead of LUT-GEMM | OPT-66B · zero-shot downstream tasks | mean accuracy: 50.70 -> 63.89 | Helped |
| Use automated mixed bit allocation (2.2-bit average) instead of fixed 2-bit allocation (ablation) | Ours (Lat.) · WikiText-2, OPT-13B | perplexity: 14.76 -> 13.79 | Helped |
| Add multi-objective optimization (ablation) | vanilla ShiftAddLLM · WikiText-2, OPT-13B, 2-bit | perplexity: 1.5e4 -> 14.76 | Helped |
| Add automated mixed bit allocation (ablation) | ShiftAddLLM with multi-objective optimization · WikiText-2, OPT-13B, 2-bit | perplexity: 14.76 -> 13.79 | Helped |
| Use multi-objective column-wise scaling instead of a weight-only objective (ablation) | ShiftAddLLM · WikiText-2, OPT-30B | perplexity: 222.6 -> 9.6 | Helped |
| Use multi-objective column-wise scaling instead of an activation-only objective (ablation) | ShiftAddLLM · WikiText-2, OPT-30B | perplexity: 10.5 -> 9.6 | Helped |
| Use multi-objective column-wise scaling instead of the vanilla weight-plus-activation objective (ablation) | ShiftAddLLM · WikiText-2, OPT-30B | perplexity: 16.3 -> 9.6 | Helped |
Surprising findings
- Naively combining weight and activation objectives performs worse than either alone (e.g., OPT-66B PPL 1178.1 vs 163.2 and 14.3 for the single objectives in the ablation), motivating the column-wise design.
- Previous uniform weight-only baselines such as AWQ at 2 bits collapse to perplexities in the thousands or more on several models.
Limitations
- GPU speedups depend on BCQ kernels and only the block-wise scaling variant (Ours Lat.) is latency-optimized; the column-wise variant (Ours Acc.) lacks a fast CUDA kernel.
- Energy figures are estimated with an Eyeriss-like accelerator model rather than measured on hardware.
- Focuses on memory-bound large LLMs; lower-bit settings are motivated mainly for that regime.
Experimental setup
Datasets: WikiText-2
Benchmarks: WikiText-2 ARC BoolQ COPA PIQA RTE StoryCloze MMLU
Metrics: perplexity zero-shot accuracy latency energy GPU memory
Baselines: OPTQ LUT-GEMM QuIP AWQ FlexRound OmniQuant MSFP
Builds on: binary-coding quantization OPTQ DenseShift Optimal Brain Compression AWQ
Closest papers by meaning
- Learning from Students: Applying t-Distributions to Explore Accurate and Efficient Formats for LLMs (ICML 2024 · similarity 0.91)
- ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models (AAAI 2025 · similarity 0.91)
- BiLLM: Pushing the Limit of Post-Training Quantization for LLMs (ICML 2024 · similarity 0.90)
- Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs (ICML 2024 · similarity 0.90)
- La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation (ICML 2025 · similarity 0.90)
- Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation (AAAI 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.