Research map / Large language models / Reasoning and chain of thought
NestedFP: High-Performance, Memory-Efficient Dual-Precision Floating Point Support for LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NestedFP, which stores FP16 weights as two 8-bit tensors so the upper part serves as an E4M3 FP8 model and both parts reconstruct FP16, enabling dual-precision LLM serving with no extra memory.
Problem
LLM serving must meet SLOs under second-level load fluctuations; running FP8 under surges helps but storing both FP16 and FP8 models costs memory, while on-the-fly quantization loses FP8 throughput.
Approach
Decompose each FP16 weight into an upper 8-bit part (a quantization-aware E4M3-like FP8 representation, exploiting the unused most significant exponent bit for weights with |w|<=1.75) and a lower 8-bit part; FP8 mode loads only the upper part, FP16 mode reconstructs losslessly via a custom CUTLASS-based GEMM kernel with merged bitwise ops and a three-stage pipeline.
Key results
- FP8 mode of NestedFP matches baseline FP8 accuracy across six models (e.g., Llama 3.1 8B Minerva Math 17.2 vs 17.6, MMLU Pro 33.2 vs 32.8).
- FP16 GEMM kernel incurs 6.38% average overhead relative to CUTLASS FP16 baseline; end-to-end FP16 overhead 4.56-5.56% across four models.
- On 4xH100 with Llama 3.1 70B, at 7.47 req/s tight SLO attainment rises from 0.7% (FP16) to 72.4% (NestedFP).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use NestedFP FP8 extracted from FP16 weights instead of baseline FP8 quantization | LLM models · Minerva Math, MMLU Pro, and BBH | accuracy: Llama 3.1 8B Minerva Math: 17.6 -> 17.2; MMLU Pro: 32.8 -> 33.2; BBH: 38.6 -> 39.0 | Mixed |
| Switch dynamically from FP16 to FP8 when token count exceeds 1024 | Llama 3.1 70B serving · Azure trace, 7.47 req/s | tight SLO attainment: 0.7 -> 72.4 | Helped |
| Switch dynamically from FP16 to FP8 when token count exceeds 1024 | Llama 3.1 70B serving · Azure trace, 5.03 req/s | tight SLO attainment: 71.6 -> 94.6 | Helped |
| Switch dynamically from FP16 to FP8 when token count exceeds 1024 | Llama 3.1 70B serving · Azure trace, 3.79 req/s | tight SLO attainment: 83.1 -> 96.6 | Helped |
| Switch dynamically from FP16 to FP8 when token count exceeds 1024 | Llama 3.1 70B serving · Azure trace, 1.99 req/s | tight SLO attainment: 90.4 -> 98.4 | Helped |
| Use NestedFP FP16 reconstruction GEMM instead of CUTLASS FP16 GEMM | FP16 GEMM inference · 14 GEMM shapes | relative performance overhead: 6.38% | Hurt |
| Use NestedFP FP16-mode execution instead of vanilla vLLM FP16 | Llama 3.1 8B · End-to-end inference across batch sizes | throughput overhead: 5.04% | Hurt |
| Use NestedFP FP16-mode execution instead of vanilla vLLM FP16 | Mistral Nemo · End-to-end inference across batch sizes | throughput overhead: 5.56% | Hurt |
| Use NestedFP FP16-mode execution instead of vanilla vLLM FP16 | Phi-4 · End-to-end inference across batch sizes | throughput overhead: 4.76% | Hurt |
| Use NestedFP FP16-mode execution instead of vanilla vLLM FP16 | Mistral Small · End-to-end inference across batch sizes | throughput overhead: 4.56% | Hurt |
| Use FP8 instead of FP16 inference | Llama 3.1 8B · Azure trace, 20% load over 60 seconds | SLO violations: 35.8% -> 17.3% | Helped |
Surprising findings
- NestedFP's FP8 mode sometimes slightly exceeds the baseline FP8 accuracy (e.g., Phi-4 BBH 28.7 vs 27.1).
Limitations
- Layers with weight magnitudes above 1.75 remain FP16-only, limiting FP8 benefit (e.g., 8.75% of Phi-4 layers; Gemma 3 only ~76-82% applicable).
- FP16 mode incurs a kernel overhead of about 6% versus CUTLASS.
- SLO evaluation uses a simple threshold switching policy on a single model/setup.
Experimental setup
Datasets: Azure LLM inference trace Minerva Math MMLU Pro BBH
Benchmarks: Minerva Math MMLU Pro BBH
Metrics: accuracy GEMM latency overhead end-to-end throughput p90 TTFT p90 TPOT SLO attainment
Baselines: vanilla vLLM FP16 CUTLASS FP16 GEMM FP8 baseline per-channel/per-token quantization FP8-only serving separate storage dual model
Builds on: CUTLASS vLLM E4M3 FP8 format LM Evaluation Harness
Closest papers by meaning
- Pushing the Limits of BFP on Narrow Precision LLM Inference (AAAI 2025 · similarity 0.89)
- Learning from Students: Applying t-Distributions to Explore Accurate and Efficient Formats for LLMs (ICML 2024 · similarity 0.87)
- Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference (NeurIPS 2025 · similarity 0.86)
- µnit Scaling: Simple and Scalable FP8 LLM Training (ICML 2025 · similarity 0.85)
- Efficient Multi-round LLM Inference over Disaggregated Serving (ICML 2026 · similarity 0.85)
- Towards Fully FP8 GEMM LLM Training at Scale (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.