Research map / Architectures and efficiency / Quantization
Scaling Laws for Floating–Point Quantization Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Capybara, a unified scaling law for floating-point quantized LLM training that jointly models data size, model size, exponent bits, mantissa bits, and scaling-factor block size, fit on 366 runs of LLaMA-style models from 41M to 679M parameters.
Problem
Existing precision scaling laws were built for integer quantization (bit width as precision) and fit poorly for floating-point quantized LLM training, where exponent, mantissa and scaling-block size have distinct effects on loss.
Approach
Extends the Chinchilla law with a multiplicative term D^beta/N^alpha * log2(B) / (gamma (E+0.5)^delta (M+0.5)^nu) capturing how exponent, mantissa and block size add quantization loss scaled by "knowledge intensity"; derives optimal exponent-mantissa layouts, a critical data size, and compute-optimal precision.
Key results
- The fitted Capybara law (Eq. with fitted constants n=69.2, alpha=0.2368, beta=0.5162, gamma=11334.5, delta=3.19, nu=2.95) fits training runs better than the Kumar et al. precision-aware law, which shows considerable bias in the E1M1 case.
- Exponent bits contribute slightly more to performance than mantissa bits; optimal layouts for FP4, FP8, FP16 are E2M1, E4M3, E8M7.
- A 1B BF16 model has a critical data size of about 1730T tokens, falling to 27T for FP8-E4M3 and 0.4T for FP4-E2M1.
- Estimated cost-performance optimal precision lies between 4 and 8 bits for compute budgets of 1e21 to 1e31 FLOPs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Capybara instead of earlier precision-aware scaling laws to model FP quantized training | LLM training-loss prediction under FP quantization · Low-precision training settings | fit to observed loss | Helped |
| Quantize P5, the input to the parameter-gradient computation (ablation) | Transformer FP3 quantization · LLM pretraining | loss: up to 2% | Hurt |
| Quantize P1, P3, or P5 (ablation) | Transformer FP3 quantization · LLM pretraining | loss | Hurt |
| Quantize only P4 or only P6 (ablation) | Transformer FP3 quantization · LLM pretraining | loss | Helped |
| Quantize P2 and P6 together instead of P2 alone (ablation) | Transformer FP3 quantization · LLM pretraining | loss | No clear effect |
| Quantize P2, P4, and P6 together instead of P2 alone (ablation) | Transformer FP3 quantization · LLM pretraining | loss | No clear effect |
| Increase training data beyond the critical data size | Low-precision LLM training · Data sizes exceeding the critical threshold | validation loss | Hurt |
| Increase exponent bits relative to mantissa bits | FP quantization training · Different bit widths | model performance | Helped |
| Use the derived optimal exponent-mantissa allocation | FP quantization training · FP4, FP8, and FP16 | loss | Helped |
| Increase model size or training precision | Low-precision LLM training · Critical data size | critical data size | Helped |
| Use precision in the 4-8 bits range under the analyzed compute budgets | FP quantization training · Compute budgets from (10^{21}, 10^{31}) FP operations | cost-performance precision: 4-8 bits | Helped |
Surprising findings
- Beyond a critical data size, adding more training data degrades loss under low-precision training.
- Quantizing P5 (backward input-embedding gradient path) raises loss by up to 2%, while quantizing P4 or P6 alone yields near-optimal performance.
Limitations
- Experiments limited to small models (41M-679M) with validation via 1.2B, 7B and 70B only as predicted points; the authors note they plan to verify larger scales.
- Only the classical Transformer (LLaMA-style) architecture and simulated FP formats with nearest rounding via QPyTorch were studied.
- Only classical FP quantization strategies were covered, not newer low-bit methods or hardware formats.
Experimental setup
Datasets: Dolma V1.7 (subset)
Metrics: validation loss fitting accuracy of scaling law
Baselines: Chinchilla scaling law OpenAI scaling law Precision-aware scaling law (Kumar et al.)
Builds on: Chinchilla scaling law Precision-aware scaling laws (Kumar et al.) OLMo
Closest papers by meaning
- ScaleBITS: Scalable Bitwidth Search for Hardware-Aligned Mixed-Precision LLMs (NeurIPS 2026 · similarity 0.89)
- Pushing the Limits of BFP on Narrow Precision LLM Inference (AAAI 2025 · similarity 0.88)
- Optimizing Large Language Model Training Using FP4 Quantization (ICML 2025 · similarity 0.88)
- FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmetic (NeurIPS 2025 · similarity 0.88)
- QuIP$\#$: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks (ICML 2024 · similarity 0.88)
- BiE: Bi-Exponent Block Floating-Point for Large Language Models Quantization (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.