Research map / Architectures and efficiency / RNNs and normalization
Scaling Laws for Precision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes precision-aware scaling laws that model low-precision training as reducing effective parameter count, predicting both quantization-aware/low-precision training loss and post-training quantization degradation in one functional form.
Problem
Current scaling laws ignore numerical precision, though training and serving precision strongly affect the cost and quality of language models; it is unclear how precision, parameters, and data trade off during pretraining and inference.
Approach
Treat weight, activation, and KV-cache bit precision as multiplicatively reducing an effective parameter count N_eff, substitute N_eff into the Chinchilla loss form, and model post-training quantization degradation as a function of data/parameter ratio, training precision, and post-training precision, yielding a unified law.
Key results
- Post-training quantization degradation grows with tokens/parameter ratio as D^gamma_D / N^gamma_N, so overtrained models can get worse after quantization past a critical data size.
- Compute-optimal pretraining precision is roughly independent of compute, estimated at about 7-8 bits when fit on integer-type quantization; with N fixed, optimal precision grows as log C.
- The unified functional form fits all runs with R^2 = 0.90 on over 1000 post-quantized data points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase pretraining data before post-training quantization | BF16-or-higher language models · Post-training quantization | Loss degradation | Hurt |
| Increase model size at a fixed dataset size | BF16-or-higher language models · Post-training quantization | Loss degradation | Helped |
| Lower the post-training weight precision | BF16-trained language models · Post-training quantization | Loss degradation | Hurt |
| Train weights, activations, and KV cache at independently chosen precisions using a multiplicative effective-parameter model | Precision-aware training scaling law · Language-model pretraining | Predictive power | Helped |
| Increase training precision from very low values | Quantized language-model pretraining · Pretraining validation loss | Loss | Helped |
| Increase training precision further after roughly 6-7 bits per weight | Quantized language-model pretraining · Pretraining validation loss | Loss | No clear effect |
| Train at lower precision before post-training quantization | Language models · Post-training quantization | Loss degradation | Helped |
| Jointly optimize model size, data, and precision rather than fixing precision | Compute-optimal pretraining · Fixed compute budget | Compute-optimal training precision: 7-8 bits | Helped |
| Use alternative hyperparameters and train on C4 (ablation) | Quantized language-model pretraining · C4 | Pretraining validation loss trend across precision | No clear effect |
| Replace the N_eff functional form with an additive/independent power law (ablation) | Precision-aware scaling law · Held-out validation | Val R²: 0.82 -> 0.71 | Hurt |
| Use N_eff and D_eff without tying them (ablation) | Precision-aware scaling law · Held-out validation | Val R²: 0.82 -> 0.84 | Helped |
| Replace the cosine learning-rate schedule with warmup and linear decay (ablation) | 30M language model · Post-training quantization across token/parameter ratios | Loss degradation trend | No clear effect |
Surprising findings
- Training in lower precision makes models more robust to post-training quantization, which outweighs the reduced effective parameter count effect.
- Overtrained models can get worse loss after post-training quantization as pretraining data increases, making extra pretraining data harmful at inference.
Limitations
- Uses a fixed architecture and small model scale (up to 1.7B parameters, 26B tokens), so only loss is studied without downstream evaluation.
- Fake/simulated quantization does not capture real low-precision compute speedups; systems overhead makes gains less than linear.
- Fitted constants vary with implementation details and integer vs floating-point type, so numerical values are suggestive rather than prescriptive.
Experimental setup
Datasets: Dolma V1.7 C4
Metrics: validation loss R^2 degradation delta_PTQ
Baselines: GPTQ AWQ round-to-nearest (RTN) Chinchilla scaling law
Builds on: Chinchilla scaling laws GPTQ depth-muP
Closest papers by meaning
- Resolving Discrepancies in Compute-Optimal Scaling of Language Models (NeurIPS 2024 · similarity 0.93)
- Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.92)
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.92)
- Scaling Inference-Efficient Language Models (ICML 2025 · similarity 0.92)
- Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs (ICLR 2026 · similarity 0.91)
- Observational Scaling Laws and the Predictability of Langauge Model Performance (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.