Research map / Large language models / Reasoning and chain of thought
Learning from Students: Applying t-Distributions to Explore Accurate and Efficient Formats for LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Profiles 30 DNN weight/activation distributions as Student's t, derives the Student Float (SF4) lookup datatype, and proposes supernormal E2M1 and APoT4 variants, mapping an accuracy-area Pareto frontier across 11 four-bit-or-so datatypes.
Problem
Choosing 4-bit quantization datatypes that match LLM weight and activation distributions while keeping MAC unit hardware cost low; existing options trade accuracy (lookup formats like NF4) against hardware efficiency (INT4).
Approach
Most DNN weights and activations fit Student's t-distributions with small degrees of freedom; deriving a quantile-based datatype for t(nu=5) gives SF4, and adding "supernormal" values to E2M1 (reusing the negative zero code) yields hardware-cheap formats with higher accuracy.
Key results
- SF4 improves LLaMA2-7B average accuracy over NF4 by 0.76% across tasks (weight-only, subchannel 128).
- E2M1 with super-precision increases Phi-2 accuracy by up to 2.19% with 1.22% estimated chip area overhead.
- E2M1 reduces average accuracy drop versus INT4 by 7.34% at about 0.6% system overhead.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace NF4 with SF4 for weight-only quantization | LLaMA2-7B · Weight-only zero-shot tasks | average relative accuracy change from FP32: -1.10 -> -0.22 | Helped |
| Add super-precision support to E2M1 (ablation) | Phi-2 · Weight-only LAMBADA | accuracy: 59.95 -> 61.73 | Helped |
| Add super-precision support to E2M1 (ablation) | Phi-2 · Weight-only WikiText-2 | perplexity: 11.97 -> 11.92 | Helped |
| Add super-range support to E2M1 (ablation) | Phi-2 · Weight-only WikiText-2 | perplexity: 11.97 -> 12.58 | Hurt |
| Add super-precision support to E2M1 (ablation) | LLaMA2-7B · Weight-only zero-shot tasks | average relative accuracy change from FP32: -0.85 -> -0.80 | Helped |
| Add super-precision support to APoT4 (ablation) | LLaMA2-7B · Weight-only zero-shot tasks | average relative accuracy change from FP32: -0.86 -> -0.39 | Helped |
| Set SF4 degrees of freedom to 5 rather than 3 (ablation) | OPT-125M · LAMBADA | accuracy: 37.18 -> 38.56 | Helped |
| Set SF4 degrees of freedom to 5 rather than 6 (ablation) | Phi-2 · LAMBADA | accuracy: 60.92 -> 61.56 | Helped |
| Quantize vision model weights and activations with SF4 instead of NF4 | ResNet18 · ImageNet | accuracy: 58.04 -> 63.12 | Helped |
| Use INT4 rather than E2M1 for the MAC datatype | MAC unit · Hardware synthesis | area: 170.4 -> 160.7 | Helped |
Surprising findings
- INT4 has the lowest MAC area but the lowest accuracy, so accuracy and hardware efficiency trade off along a Pareto curve rather than one format dominating.
- Super-range E2M1 often degrades accuracy substantially (e.g., BLOOM-7B LAMBADA drops to 50.18) despite extending range.
Limitations
- Hardware cost is estimated via MAC synthesis and an assumed chip area breakdown rather than full accelerator implementation.
- SF4 is a lookup format requiring custom hardware support, so it is a reference rather than a deployable datatype.
- Accuracy gains are not uniform; many exceptions where baseline formats match or beat supernormal variants.
Experimental setup
Datasets: LAMBADA WikiText-2 HellaSwag WinoGrande PIQA BoolQ ARC-c ImageNet
Benchmarks: LAMBADA WikiText-2 HellaSwag WinoGrande PIQA BoolQ ARC-c ImageNet
Metrics: accuracy perplexity relative accuracy change MAC area power
Baselines: NF4 INT4 E2M1-I E2M1-B E3M0 APoT4 SmoothQuant GPTQ
Builds on: NF4 APoT SmoothQuant GPTQ bitsandbytes FP4
Closest papers by meaning
- ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization (NeurIPS 2024 · similarity 0.91)
- Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference (NeurIPS 2025 · similarity 0.90)
- Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs (ICML 2024 · similarity 0.90)
- BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Efficient LLM Inference (ICML 2025 · similarity 0.89)
- Adaptive Draft-Verification for Efficient Large Language Model Decoding (AAAI 2025 · similarity 0.89)
- DP-LLM: Runtime Model Adaptation with Dynamic Layer-wise Precision Assignment (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.