Research map / Architectures and efficiency / Quantization
Pushing the Limits of BFP on Narrow Precision LLM Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DB-Attn, a hardware-software co-design that applies an adaptive block floating-point variant (DBFP) and a hierarchical LUT (DH-LUT) to nonlinear Softmax in attention, with an RTL engine for FPGA/ASIC.
Problem
Nonlinear operations such as Softmax in LLM attention remain in floating-point and become bottlenecks as sequence length grows; vanilla block floating-point (BFP) loses accuracy when applied to them due to outlier-driven exponent alignment and coarse granularity.
Approach
DBFP aligns exponents to a median-like pivot rather than the group maximum (pivot-focus) and groups elements by exponent interval (adaptive grouping); DH-LUT uses a non-uniform hierarchical lookup table over DBFP data so exponentials can be computed in a BFP-like, integer-friendly format.
Key results
- On LLaMA3-8B, DBFP+DH-LUT matches FP16 zero-shot average (74.26 vs 74.17) and WikiText2 perplexity (6.14), while vanilla BFP drops zero-shot average by 33.81%.
- GPU CUDA Softmax emulation on A800 reduces latency by 74% on average across LLaMA series, and at least 30% across models.
- On FPGA at 1024-length, the design reports 10x FOM over ISCAS'23 (509.563 vs 49.056) and 62.5% latency reduction versus Hyft16 baseline comparisons.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DBFP with DH-LUT for Attention nonlinear operations | LLaMA models · LLaMA series Softmax on GPU | latency: 74% | Helped |
| Use DBFP-based Softmax | diverse model architectures · GPU Softmax | speed: at least 30% | Helped |
| Align exponent to the maximum rather than the median (ablation) | Softmax BFP · Softmax | loss: 9.6x | Hurt |
| Add pivot-focus alignment using the median exponent (ablation) | DBFP Softmax · Softmax | accuracy | Helped |
| Add adaptive grouping based on exponent intervals (ablation) | DBFP · Nonlinear operations | accuracy | Helped |
| Vary LUT bit width to 5-7 bits (ablation) | DB-Attn Softmax · Softmax accuracy, computational error, and LUT memory | trade-off: 5-7 | Helped |
| Use DBFP in a 4-level adder tree | DBFP adder tree · 4-level adder tree | latency: 42% | Helped |
| Use the DBFP Divider instead of an FP16 divider | Softmax division · Softmax | latency: 83% | Helped |
| Use DBFP with DH-LUT instead of vanilla BFP | LLaMA models · Zero-shot tasks | accuracy | Helped |
| Use DBFP-based Softmax accelerator at input length 1024 | FPGA Softmax accelerator · Softmax accelerator FPGA | FOM: 10x | Helped |
Surprising findings
- Setting the BFP alignment pivot to the group maximum causes up to 9.6x greater Softmax loss than using the median.
- FP8 e4m3 collapses to near-random zero-shot accuracy on LLaMA models because it cannot represent infinity.
Limitations
- GPU evaluation is an emulation via custom CUDA operator rather than native BFP hardware.
- ASIC comparison is normalized to 28nm and FPGA comparisons use differing bit widths and input sizes.
- Evaluation limited to LLaMA-family LLMs at 7-8B scale and small vision models.
Experimental setup
Datasets: WikiText2 C4 PIQA ARC-e ARC-c BoolQ HellaSwag Winogrande ImageNet COCO
Benchmarks: WikiText2 C4 PIQA ARC-e ARC-c BoolQ HellaSwag Winogrande ImageNet COCO
Metrics: perplexity zero-shot accuracy top-1 accuracy mAP latency LUT FF FOM ADP EDP
Baselines: FP8 e4m3 FP8 e5m2 vanilla BFP Hyft TCAS-I'22 ISCAS'23 TCAS-II'22 Xilinx FP IP
Builds on: block floating point MXFP BSFP
Closest papers by meaning
- NestedFP: High-Performance, Memory-Efficient Dual-Precision Floating Point Support for LLMs (NeurIPS 2025 · similarity 0.89)
- Learning from Students: Applying t-Distributions to Explore Accurate and Efficient Formats for LLMs (ICML 2024 · similarity 0.89)
- Scaling Laws for Floating–Point Quantization Training (ICML 2025 · similarity 0.88)
- ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models (AAAI 2025 · similarity 0.88)
- Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference (NeurIPS 2025 · similarity 0.88)
- Puzzle: Distillation-Based NAS for Inference-Optimized LLMs (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.