Research map / Architectures and efficiency / Quantization
70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DFloat11, a lossless BFloat16 compression format that Huffman-codes the exponent field and uses a hierarchical-LUT, two-phase GPU decompression kernel to run LLMs and diffusion models at ~70% size with bit-identical outputs.
Problem
BFloat16 LLM and diffusion model weights are large (e.g., Llama 3.1 405B at 810GB), and existing lossy quantization degrades behavior while existing lossless methods only help storage or require non-GPU hardware or nvCOMP, so efficient lossless GPU inference is lacking.
Approach
Entropy-code only the low-entropy 8-bit BFloat16 exponent with Huffman coding, keep sign and mantissa raw, and decode on GPU using compact hierarchical 256-entry lookup tables in SRAM, a two-phase kernel with gap array and block-level output positions, and batched per-transformer-block decompression.
Key results
- DF11 compresses tested LLMs and diffusion transformers to about 67.6-69.5% of original size (~10.8-11.1 average bits), e.g., Llama 3.1 405B from 811.71 GB to 551.22 GB.
- Llama 3.1 8B Instruct DF11 matches BF16 exactly on MMLU (68.010) and TruthfulQA (36.965) and on WikiText/C4 perplexity (8.649/21.677).
- Versus BF16 with CPU offloading, DF11 achieves 2.31-46.24x lower latency or higher throughput, and enables 5.70-14.86x longer generation under fixed GPU memory.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Compress BF16 weights losslessly with DFloat11 | Llama 3.1 8B Instruct · MMLU | accuracy: 68.010 ± 0.375 -> 68.010 ± 0.375 | No clear effect |
| Compress BF16 weights losslessly with DFloat11 | Llama 3.1 8B Instruct · TruthfulQA | accuracy: 36.965 ± 1.690 -> 36.965 ± 1.690 | No clear effect |
| Compress BF16 weights losslessly with DFloat11 | Llama 3.1 8B Instruct · WikiText | perplexity: 8.649 -> 8.649 | No clear effect |
| Compress BF16 weights losslessly with DFloat11 | Llama 3.1 8B Instruct · C4 | perplexity: 21.677 -> 21.677 | No clear effect |
| Compress BF16 weights losslessly with DFloat11 | Stable Diffusion 3.5 Large · 1024×1024 image generation | peak GPU memory: reduces memory consumption by 28.3% | Helped |
| Compress BF16 weights losslessly with DFloat11 | Stable Diffusion 3.5 Large · 1024×1024 image generation | generation latency: relative increase in latency is small: 4.1% | Hurt |
| Compress BF16 weights losslessly with DFloat11 | FLUX.1 dev · 1024×1024 image generation | peak GPU memory: reduces memory consumption by 27.8% | Helped |
| Compress BF16 weights losslessly with DFloat11 | FLUX.1 dev · 1024×1024 image generation | generation latency: relative increase in latency is small: 5.5% | Hurt |
| Use DFloat11 instead of BF16 with partial CPU offloading | LLM inference · token generation | throughput: 2.31--46.24× higher throughput | Helped |
| Use DFloat11 instead of BF16 under a fixed GPU memory budget | LLM inference · generation before reaching memory limits | generation length: 5.70 to 14.86× more tokens | Helped |
| Use DFloat11 decompression instead of CPU-to-GPU weight transfer | Llama 3.1 8B Instruct language modeling head · sliced weight matrices | throughput: up to 34.95× higher throughput | Helped |
| Use DFloat11 decompression instead of nvCOMP ANS | Llama 3.1 8B Instruct language modeling head · sliced weight matrices | decompression speed: up to 20.97× faster decompression | Helped |
Surprising findings
- Only about 40 of the 256 possible 8-bit exponent values are used in LLM weights, and the exponent carries only about 2.6 bits of entropy.
- DF11 decompression is up to 20.97x faster than nvCOMP ANS, and nvCOMP ANS gives worse compression (79%) than DF11 (68%).
Limitations
- Small but non-zero latency overhead from decompression, most noticeable at small batch sizes.
- Only BFloat16 weights are handled; FP32, FP16 and FP8 are not considered.
- Compression of 405B takes about 2133 s per transformer block on a single CPU thread (preprocessing cost).
Experimental setup
Datasets: MMLU TruthfulQA WikiText C4
Benchmarks: MMLU TruthfulQA WikiText C4
Metrics: accuracy perplexity compression ratio throughput latency peak GPU memory maximum generation length
Baselines: BF16 with CPU offloading nvCOMP ANS CPU-to-GPU transfer NeuZip INT8 quantization (SmoothQuant/GPTQ context)
Builds on: Huffman coding Deep Compression nvCOMP ANS HuggingFace Transformers Flash Attention
Closest papers by meaning
- SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (ICLR 2024 · similarity 0.91)
- Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference (ICML 2024 · similarity 0.90)
- Speculate Deep and Accurate: Lossless and Training-Free Acceleration for Offloaded LLMs via Substitute Speculative Decoding (NeurIPS 2025 · similarity 0.90)
- Practical Offloading for Fine-Tuning LLM on Commodity GPU via Learned Sparse Projectors (AAAI 2025 · similarity 0.90)
- SpeedLoader: An I/O efficient scheme for heterogeneous and distributed LLM operation (NeurIPS 2024 · similarity 0.89)
- Cost-Efficient LLM Training with Lifetime-Aware Tensor Offloading via GPUDirect Storage (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.