Research map / Architectures and efficiency / Quantization
BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Efficient LLM Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BlockDialect, a block-wise mixed-format 4-bit quantization that picks one of 16 FP4 variants ("dialects") per 32-element block with a two-stage online selection, enabling integer MAC arithmetic for both weights and activations.
Problem
Activation quantization for LLM inference is hard (real-time, wide dynamic range, channel outliers), and existing fine-grained hardware-supported formats like MXFP4 fail to capture nuanced per-block value distributions while keeping energy-efficient low-precision arithmetic.
Approach
Instead of only scaling each block, assign each block its own FP4 variant from a 16-entry formatbook (DialectFP4) whose values are multiples of 0.5 so products are integer ops; select the dialect online with a two-stage process (max-magnitude pair, then beneficial-range counting) rather than MSE.
Key results
- On LLaMA3-8B, BlockDialect with full-path quantization is 10.78% higher accuracy than MXFP4-16 with lower bit usage and 5.45% below FP16 (LLaMA2-7B: 7.48% higher, 2.69% below).
- In the linear scope on LLaMA3-8B, BlockDialect-32 reaches 72.24 average zero-shot accuracy vs. 68.31 for MXFP4-32 at 4.28 vs 4.16 effective bits, with 1.76% drop from FP16 at the mixed configuration.
- The DialectFP4 MAC unit is 1.58x more power-efficient and 1.54x more area-efficient than FP6 MACs and 2.45x / 2.23x more efficient than INT8 MACs (45nm synthesis).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BlockDialect-32 instead of MXFP4-32 | LLaMA3-8B linear quantization · WikiText2 | perplexity: 8.23 -> 7.05 | Helped |
| Use BlockDialect-32 instead of MXFP4-32 | LLaMA3-8B linear quantization · Commonsense 0-shot average | accuracy: 68.31 -> 72.24 | Helped |
| Extend quantization from linear layers to all matrix multiplications | BlockDialect-32 · LLaMA3-8B | accuracy: 72.24 -> 68.57 | Hurt |
| Use smaller blocks in quantization-sensitive sublayers (dn, Q, K) (ablation) | BlockDialect-64, all matrix multiplications · LLaMA3-8B | accuracy: 66.60 -> 69.00 | Helped |
| Reduce block size from 32 to 16 (ablation) | BlockDialect, linear quantization · LLaMA3-8B | accuracy: 72.24 -> 72.98 | Helped |
| Use 8 dialects instead of 16, prioritizing magnitude distribution (ablation) | BlockDialect, all matrix multiplications · LLaMA3-8B | accuracy: 68.57 -> 67.96 | Hurt |
| Use 24 dialects instead of 16 (ablation) | BlockDialect, all matrix multiplications · LLaMA3-8B | accuracy: 68.57 -> 67.57 | Hurt |
| Prioritize maximum magnitude rather than magnitude distribution among 8 dialects (ablation) | BlockDialect, all matrix multiplications · LLaMA3-8B | accuracy: 67.96 -> 68.06 | Helped |
| Replace two-stage dialect selection with exact MSE selection (ablation) | BlockDialect-32, linear quantization · LLaMA3-8B | accuracy: 72.24 -> 72.85 | Helped |
| Replace 1D blocks with 2D blocks (ablation) | BlockDialect-32, linear quantization · LLaMA3-8B | perplexity: 7.05 -> 7.09 | Hurt |
| Add SmoothQuant (ablation) | BlockDialect-64, linear quantization · LLaMA3-8B | accuracy: 71.51 -> 71.54 | No clear effect |
Surprising findings
- Two-stage online dialect selection without MSE computation performs within about 0.6% accuracy of MSE-based selection, while using much less hardware (9.32x smaller than MSE logic).
Limitations
- Evaluation is emulation-based on PyTorch/HuggingFace; no end-to-end hardware speedup measured.
- Gains are limited when combined with SmoothQuant in some models, and interaction with rotation-based methods is not fully explored.
- Smaller blocks increase effective bitwidth, so accuracy comes with a memory-footprint trade-off.
Experimental setup
Datasets: WikiText2
Benchmarks: WikiText2 LAMBADA HellaSwag BoolQ PIQA WinoGrande ARC-easy ARC-challenge MMLU GLUE
Metrics: perplexity zero-shot accuracy effective bitwidth area power
Baselines: MXFP4 LLM-FP4 QuaRot NVFP4 MSE-based dialect selection
Builds on: MXFP4 Microscaling (MX) format QuaRot LLM-FP4 SmoothQuant
Closest papers by meaning
- ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models (AAAI 2025 · similarity 0.89)
- Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edge (AAAI 2024 · similarity 0.89)
- Learning from Students: Applying t-Distributions to Explore Accurate and Efficient Formats for LLMs (ICML 2024 · similarity 0.89)
- RaZeR: Pushing the Limits of NVFP4 Quantization with Redundant Zero Remapping (NeurIPS 2026 · similarity 0.89)
- CodeGEMM: A Codebook-Centric Approach to Efficient GEMM in Quantized LLMs (NeurIPS 2025 · similarity 0.89)
- any4: Learned 4-bit Numeric Representation for LLMs (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.