Research map / Large language models / Language, safety and interpretability
any4: Learned 4-bit Numeric Representation for LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes any4, a learned per-row 4-bit weight codebook (k-means-style weighted by activation statistics from one hand-written calibration prompt) for LLM weight-only PTQ, plus the tinygemm LUT-based GEMM library.
Problem
Choosing the 4-bit numeric representation for weight-only LLM quantization: int4, fp4 and nf4 are fixed formats that may not match the actual weight distribution, and strong accuracy often requires preprocessing (AWQ, GPTQ, Hadamard transforms) or large calibration sets.
Approach
Learn an arbitrary 16-entry lookup table per row of each weight matrix via weighted k-means over group-scaled weights, with weights set by the mean absolute input activation per channel, so no weight or activation preprocessing is needed and calibration uses a single handwritten prompt.
Key results
- On Llama3 70B, any4 reaches WikiText-2 perplexity 3.20 vs 3.43 for NF4 and 3.63 for INT4 (FP16 2.86).
- On Llama3 8B, any4 RTN 4-bit matches the best GPTQ/AWQ/QuIP int4 results (WikiText-2 6.5) without preprocessing.
- Calibrating on a single 88-word handwritten prompt gives C4 perplexity 13.95 on Llama3.2 1B, better than using 128 C4 samples of 2048 tokens (14.05).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use any4 rather than nf4 for 4-bit quantization | Llama3.2 1B · WikiText-2 | perplexity: 10.99 -> 10.63 | Helped |
| Use any4 rather than nf4 for 4-bit quantization | Llama3 8B · MMLU | accuracy: 60.7% -> 61.0% | Helped |
| Use any4 rather than GPTQ INT4 | Llama3 70B · WikiText-2 | perplexity: 3.3 -> 3.2 | Helped |
| Use one handwritten prompt for calibration instead of 128 C4 samples of sequence length 2048 (ablation) | any4 on Llama3.2 1B · C4 | perplexity: 14.05 -> 13.95 | Helped |
| Calibrate on Pile instead of C4 (ablation) | any4 on Llama3.2 1B · C4 | perplexity: 14.05 -> 13.99 | Helped |
| Calibrate on C4 with 512-token sequences instead of 2048-token sequences (ablation) | any4 on Llama3.2 1B · C4 | perplexity: 14.05 -> 13.96 | Helped |
| Increase quantization group size from 128 to 1024 (ablation) | any4 on Llama3.2 1B · C4 | perplexity: 13.95 -> 14.34 | Hurt |
| Decrease quantization group size from 128 to 64 (ablation) | any4 on Llama3.2 1B · C4 | perplexity: 13.95 -> 13.75 | Helped |
| Include activations in the LUT minimization objective rather than minimizing weights only (ablation) | any4 on Llama3.2 1B · WikiText-2 | perplexity: 6.680 -> 6.496 | Helped |
| Include group scales in the objective alongside weights and activations (ablation) | any4 on Llama3.2 1B · WikiText-2 | perplexity: 6.496 -> 6.487 | Helped |
| Initialize k-means with random centroids rather than k-means++ (ablation) | any4 on Llama3.2 1B · WikiText-2 | perplexity: 10.63 -> 10.66 | Hurt |
| Initialize k-means with uniform int4 values rather than k-means++ (ablation) | any4 on Llama3.2 1B · WikiText-2 | perplexity: 10.63 -> 10.83 | Hurt |
Surprising findings
- A single 88-word handwritten prompt calibrates as well or better than 128 long sequences from C4 or Pile.
- any4 with a single LUT per row achieves nearly the same speedup as nf4, which uses one LUT per tensor.
Limitations
- Orthogonal preprocessing techniques (AWQ, GPTQ, Hadamard) are not combined with any4 in the paper.
- Downstream task accuracies are noisy; perplexity is the main signal.
- Kernel speedups are lower than int4 due to LUT lookup overhead; tinygemm targets Ampere and later GPUs with small batch sizes.
Experimental setup
Datasets: handwritten calibration prompt WikiText-2 C4 Penn Treebank CodeParrot
Benchmarks: WikiText-2 C4 PTB CodeParrot HumanEval MBPP MMLU HellaSwag GSM8K BBH BigBench
Metrics: perplexity pass@1 accuracy latency speedup
Baselines: INT4 (RTN) FP4 NF4 GPTQ AWQ QuIP SqueezeLLM
Builds on: NormalFloat4 (nf4) k-means clustering AWQ GPTQ QuIP
Closest papers by meaning
- Optimizing Large Language Model Training Using FP4 Quantization (ICML 2025 · similarity 0.90)
- QuEST: Stable Training of LLMs with 1-Bit Weights and Activations (ICML 2025 · similarity 0.89)
- OneBit: Towards Extremely Low-bit Large Language Models (NeurIPS 2024 · similarity 0.89)
- BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Efficient LLM Inference (ICML 2025 · similarity 0.89)
- Norm Tweaking: High-Performance Low-Bit Quantization of Large Language Models (AAAI 2024 · similarity 0.88)
- Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.