Research map / Architectures and efficiency / Quantization
CBQ: Cross-Block Quantization for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CBQ, a cross-block reconstruction post-training quantization method for LLMs that combines overlapping-window cross-block dependency, LoRA-Rounding, and coarse-to-fine outlier pre-processing to reach W4A4/W4A8/W2A16 accuracy above prior PTQ methods.
Problem
Post-training quantization of LLMs still degrades sharply at ultra-low bit widths (W2A16, W4A4), and prior layer-wise or block-wise reconstruction methods ignore inter-layer and intra-layer dependencies that grow with model size and lower bit-width.
Approach
Reconstruct outputs across multiple overlapping transformer blocks in a sliding window (cross-block dependency), learn low-rank (LoRA-style) rounding compensation matrices jointly with step sizes, and detect weight and activation outliers with a quartile-based two-stage coarse-to-fine procedure.
Key results
- On LLAMA1-65B W4A4, CBQ reaches 77.69 PIQA and 76.65 HellaSwag, improving over OmniQuant (71.81 / 66.81).
- On LLAMA-7B W4A4 ablation, the full CBQ (CFP + CBQ-Recon) reaches C4 13.29 and Wiki 10.63 PPL, versus OmniQuant-level baselines in the appendix.
- CBQ quantizes a 4-bit weight-only LLAMA1-65B model in 4.3 GPU hours, versus 8.9 hours reported for OmniQuant.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CBQ instead of QLLM for W4A4 quantization | LLAMA1-65B · PIQA | accuracy: 73.56 -> 77.69 | Helped |
| Use CBQ instead of QLLM for W4A4 quantization | LLAMA1-65B · C4 perplexity | perplexity: 8.89 -> 7.52 | Helped |
| Use CBQ instead of OmniQuant for W2A16 quantization | LLAMA1-30B · C4 perplexity | perplexity: 9.02 -> 7.65 | Helped |
| Use CBQ instead of QLLM for W4A4 quantization | LLAMA2-7B · HellaSwag | accuracy: 58.45 -> 57.34 | Hurt |
| Use CBQ instead of OmniQuant for W2A16 quantization | LLAMA2-7B · C4 perplexity | perplexity: 12.72 -> 11.30 | Helped |
| Add CFP outlier preprocessing with CBQ-Recon (ablation) | LLAMA-7B W4A4 · C4 perplexity | perplexity: 1082.68 -> 13.29 | Helped |
| Add LoRA-Rounding compensation (ablation) | LLAMA-7B W4A4 · C4 perplexity | perplexity: 14.32 -> 13.29 | Helped |
| Replace AdaRounding with LoRA-Rounding (ablation) | LLAMA-7B W4A4 · C4 perplexity | perplexity: 14.56 -> 13.29 | Helped |
| Increase LoRA-Rounding rank from 5 to 7 (ablation) | LLAMA-7B W4A4 · C4 perplexity | perplexity: 13.29 -> 13.98 | Hurt |
| Add overlapping cross-block windows (2 blocks, overlap 1) (ablation) | LLAMA-7B W4A4 · C4 perplexity | perplexity: 14.57 -> 13.29 | Helped |
| Increase cross-block window from 2 blocks with overlap 1 to 4 blocks with overlap 3 (ablation) | LLAMA-7B W4A4 · C4 perplexity | perplexity: 13.29 -> 12.32 | Helped |
| Combine L2 and KLD reconstruction losses instead of L2 alone (ablation) | LLAMA-7B W4A4 · C4 perplexity | perplexity: 13.82 -> 13.29 | Helped |
Surprising findings
- KLD loss alone (13.84 C4) performs about as well as L2 alone (13.82), and the combination gives the best result, suggesting the two losses are complementary.
Limitations
- Cross-block windows increase GPU memory and time (e.g., 4-block windows need 39 GB vs 17.2 GB for single-block), trading efficiency for accuracy.
- Some results are mixed: OmniQuant beats CBQ on LLAMA1-30B W2A16 ARC-C and LLAMA1-65B W2A16 PIQA, and the appendix reports OPT-1.3B W2A16 with a CBQ C4 value far above OmniQuant's Wiki result.
Experimental setup
Datasets: C4
Benchmarks: PIQA HellaSwag ARC-C ARC-E Mutual Ethics C4 WikiText2
Metrics: accuracy perplexity Mutual MRR/Recall@1/Recall@2
Baselines: GPTQ OmniQuant QLLM RPTQ SmoothQuant OS+
Builds on: AdaRound BRECQ OmniQuant SmoothQuant GPTQ
Closest papers by meaning
- Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation (AAAI 2024 · similarity 0.92)
- QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models (ICLR 2024 · similarity 0.92)
- Norm Tweaking: High-Performance Low-Bit Quantization of Large Language Models (AAAI 2024 · similarity 0.92)
- ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models (AAAI 2025 · similarity 0.92)
- Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantization (NeurIPS 2025 · similarity 0.92)
- ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals (ICML 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.