Research map / Architectures and efficiency / Quantization
Irrational Complex Rotations Empower Low-bit Optimizers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes π-Quant, which encodes each pair of optimizer-state parameters as a single complex rotation angle built on irrational numbers (π-like constants), enabling ~3.32-bit Adam state compression without search-based quantization.
Problem
Optimizer states (Adam momentum) dominate training memory; existing low-bit optimizer state methods rely on search-based quantization requiring custom GPU kernels and fixed bit-widths.
Approach
Represents a real parameter pair (x, y) with |x+iy| bounded as a single angle θ via x+iy = e^{iθ} + e^{iπ̄θ} with an irrational π̄, solved by closed-form geometric equations with linear complexity and quantized by truncating decimal digits of the angle, giving flexible bit-widths (3.32 bits for λ=1).
Key results
- At 3.32-bit optimizer states, π-Quant on TinyLlama reaches test PPL 5.14 and average downstream 55.34, matching FP32 Adam (5.17, 55.22) and reduces memory from 19.47 GB to 11.32 GB.
- Lpmm.Adam 4-bit degrades average downstream accuracy to 54.44; Linear-Quant at 8 and 16 bits collapses with nan loss.
- On other tasks, π-Quant outperforms full-precision Adam on most reported metrics, e.g., Image accuracy 41.22 vs 40.13 for Adam.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use π-Quant at 13.28-bit instead of FP32 | TinyLlama-1.1B · Proof-pile test | Test PPL: 5.17 -> 5.15 | Helped |
| Use π-Quant at 13.28-bit instead of FP32 | TinyLlama-1.1B · PG-19 / public benchmark average | Avg.: 55.22 -> 55.45 | Helped |
| Use π-Quant at 3.32-bit instead of Lpmm.Adam (4-bit) | TinyLlama-1.1B · Proof-pile test | Test PPL: 5.41 -> 5.14 | Helped |
| Use π-Quant at 3.32-bit instead of Lpmm.Adam (4-bit) | TinyLlama-1.1B · PG-19 / public benchmark average | Avg.: 54.44 -> 55.34 | Helped |
| Use π-Quant at 6.64-bit instead of Bnb.Adam (8-bit) | TinyLlama-1.1B · Proof-pile test | Test PPL: 5.14 -> 5.16 | Hurt |
| Use π-Quant at 6.64-bit instead of Bnb.Adam (8-bit) | TinyLlama-1.1B · PG-19 / public benchmark average | Avg.: 55.34 -> 55.40 | Helped |
| Use π-Quant instead of Adam | Standard transformer · CIFAR-10 image classification | accuracy: 40.13 -> 41.22 | Helped |
| Use π-Quant instead of Adam | Standard transformer · Samsum text summarization | Blue-4: 1.54 -> 2.02 | Helped |
| Use the π-based irrational value instead of truncated rational π̄₁ = 0.001 (ablation) | π-Quant · Gaussian and uniform parameter distributions | quantization error | Helped |
| Set the integer part of π̄ to zero instead of a nonzero value (ablation) | π-Quant · Gaussian and uniform parameter distributions | quantization error | Helped |
Surprising findings
- Linear quantization at 8-bit and 16-bit diverges (nan loss) while the rotation method remains stable.
- π-Quant at low bit-width is reported to slightly exceed full-precision Adam on several tasks, which the authors attribute to the quantization acting favorably.
Limitations
- Quant/Restore overhead from computing angles adds latency compared with optimized GPU binary-search kernels.
- Theorem requires magnitudes bounded (values scaled to a range), and quantization error is bounded by decimal precision of angle.
- Experiments are limited to a 1.1B model continual pretraining and small-scale tasks; no evaluation on larger models.
Experimental setup
Datasets: PG-19 Proof-pile ARC-Challenge Hellaswag Lambada PIQA Long Range Arena (ListOps, IMDb, AAN retrieval, PathFinder, CIFAR-10) SAMSum MovieLens
Benchmarks: Proof-pile perplexity ARC-Challenge Hellaswag Lambada PIQA Long Range Arena SAMSum MovieLens
Metrics: test perplexity accuracy train loss BLEU Recall@10 NDCG@10 MRR@10 memory step time
Baselines: FP32 Adam Linear-Quant Bnb.Adam (8-bit) Lpmm.Adam (4-bit)
Builds on: Adam 8-bit optimizers (Bnb) 4-bit optimizers (Lpmm)
Closest papers by meaning
- DartQuant: Efficient Rotational Distribution Calibration for LLM Quantization (NeurIPS 2025 · similarity 0.84)
- QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs (NeurIPS 2024 · similarity 0.83)
- Scaling Laws for Floating–Point Quantization Training (ICML 2025 · similarity 0.83)
- SLiM: One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression (ICML 2025 · similarity 0.83)
- SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (ICLR 2024 · similarity 0.83)
- CommVQ: Commutative Vector Quantization for KV Cache Compression (ICML 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.