Research map / Architectures and efficiency / RNNs and normalization
Unified Scaling Laws for Compressed Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a unified scaling law for training over compressed representations where parameter efficiency (capacity) is a parametric function of the representation's Gaussian-fitting MSE, shown to factorize across compression types and used to design an RMSE-banded backward mask for sparse training.
Problem
Scaling laws have been specialized to single compression types (sparsity or quantization) and do not predict loss when training over hybrid or composite compressed representations (sparse-quantized, weights+activations, vector-quantized).
Approach
Model representation capacity as a tanh-based parametric function of the MSE obtained when fitting random Gaussian data, sharing one functional form across compression types; capacity multiplies across composite compressions, and the MSE-based view yields an improved backward masking heuristic for sparse training.
Key results
- Capacity factorizes across composite representations (e.g., sparse weights x sparse activations, sparse x quantized), with factorized-prediction errors around 10^-2 except at extreme 2-bit quantization.
- The tanh-based unified law achieves fitting error 4.7e-4 versus 4.5e-3 for the quantization scaling law of prior work and 2.1e-3 for the compression law, with the decoupled form at 4.2e-4.
- RMSE-banded backward masking gives consistently higher sparse-representation capacity than the Magnitude Pruning baseline across 10%-90% sparsity on 30M Llama, with gains above 20% in some regimes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the unified representation-capacity scaling law instead of the sparsity-specific law | Scaling-law fitting for sparsity · Sparsity fit | fitting error: $5.7\cdot 10^{-4}$ -> $4.7\cdot10^{-4}$ | Helped |
| Use the unified representation-capacity scaling law instead of the quantization-specific law | Scaling-law fitting for quantization · Quantization fit | fitting error: $4.5\cdot10^{-3}$ -> $2.1\cdot 10^{-3}$ | Helped |
| Fit compression efficiency independently for each compression instance rather than using the unified representation-capacity law | Unified representation scaling law · Sparsity fit | fitting error: $4.2 \cdot 10^{-4}$ -> $4.7\cdot10^{-4}$ | Hurt |
| Replace tanh with logistic capacity function (ablation) | Unified representation scaling law · Combined scaling-law fit | fitting error: $4.7\cdot 10^{-4}$ -> $4.9\cdot 10^{-4}$ | Hurt |
| Replace tanh with boundary-constrained Logistic (1, 0) capacity function (ablation) | Unified representation scaling law · Combined scaling-law fit | fitting error: $4.7\cdot 10^{-4}$ -> $1.1\cdot 10^{-3}$ | Hurt |
| Use Gaussian input noise instead of Logistic, Student's t, or Laplace noise to compute representation MSE (ablation) | Representation-capacity prediction · Compressed representations | representation-capacity prediction quality | Helped |
| Apply RMSE-Banded Backward Masking instead of magnitude-pruning backward masks | Sparse training · Unstructured sparsity from 10% to 90% | representation capacity | Helped |
| Preserve outliers in higher precision rather than preserving no outliers | Quantized pretraining formats · C4 pretraining | MSE-capacity trade-off | Hurt |
| Switch from INT4 to FP4 (E2M1) | 4-bit quantized training · C4 pretraining | validation loss | No clear effect |
| Factorize capacity across sparse and quantized representations rather than fitting the joint capacity directly | Sparse-quantized weights · Combined sparsity and quantization | representation-capacity approximation error: $10^{-2}$ | No clear effect |
Surprising findings
- 2-bit weight-only quantization is Pareto optimal for the architectures considered, complementing ParetoQ.
- Preserving no outliers (0%) lies on the Pareto-optimal boundary; allocating bits to the value distribution beats outlier preservation or careful grouping for pretraining.
Limitations
- Experiments limited to decoder-only Llama-style models up to 200M non-embedding parameters trained on C4 in the data-rich regime (100 tokens/param).
- The law may need specific fits for ultra-low precision (2-bit, ternary) and for VQ codebooks below 8 entries.
- The convergence theorem assumes standard smoothness/bounded-gradient assumptions and is only extended loosely to more complex representations.
Experimental setup
Datasets: C4
Benchmarks: C4 validation loss
Metrics: validation loss fitting error representation capacity rho Gaussian MSE
Baselines: Magnitude Pruning Decoupled scaling law Sparsity scaling law Quantization scaling law ParetoQ
Builds on: Chinchilla scaling law LSQ QuEST HIGGS gradual magnitude pruning
Closest papers by meaning
- Asymmetric Scaling Laws from Sparse Features (NeurIPS 2026 · similarity 0.89)
- Effective Interplay between Sparsity and Quantization: From Theory to Practice (ICLR 2025 · similarity 0.88)
- Scaling Laws for Sparsely-Connected Foundation Models (ICLR 2024 · similarity 0.88)
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.87)
- SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (ICLR 2024 · similarity 0.87)
- Power Lines: Scaling laws for weight decay and batch size in LLM pre-training (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.