Research map / Architectures and efficiency / Quantization

LAMP: Look-Ahead Mixed-Precision Inference of Large Language Models

NeurIPS 2026 · Architectures and efficiency › Quantization · approach: Post-training quantization of LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes look-ahead mixed-precision (LAMP): adaptively recompute a sparse subset of inner-product components in higher precision, chosen by the sensitivity of the following nonlinearity, applied to attention softmax in GPT-2 models.

Problem

Low-precision floating-point accumulation in transformer inference introduces rounding errors that get amplified by subsequent nonlinearities (softmax, layer norm), and existing mixed-precision methods focus on operand quantization rather than accumulation accuracy.

Approach

Formulate a sparse selection problem (LAMP) that finds the fewest components of an inner function's output to recompute more accurately so the composition's condition number stays below a threshold; derive closed-form solutions for activations, RMS layer norm, and softmax (l1-normwise relaxed to relative-threshold for FlashAttention compatibility).

Key results

• On GPT-2 XL with KQ accumulation at mu=4, LAMP reduces KL divergence by 12x, 83x, and 385x at recomputation rates of 0.3%, 1.6%, and 7.6%.
• BF16 accumulation (mu=7) with adaptive recomputation at 0.9% rate matches the KL deviation of uniform TF32 accumulation (mu=10), with effective ~8.9 mantissa bits per inner product versus 10.
• Relaxed relative-threshold LAMP for softmax recovers perplexity close to full precision (e.g., WikiText-2 17.524 low precision vs 17.454 LAMP vs 17.411 full) with about 5.7% recomputation.

Surprising findings

• Random recomputation of the same number of inner products yields no accuracy improvement, so the choice of components matters rather than the count.
• Enlarging the greedy index set by any fixed or percentage margin can fail to satisfy the componentwise LAMP objective for softmax.

Limitations

• Experiments simulate low-precision accumulation in software; no runtime or hardware measurements.
• The strict softmax LAMP requires materializing the full probability vector and is incompatible with FlashAttention; the relaxed variant is only a preliminary step.
• Validated only on GPT-2 family models (small and XL), not larger modern transformers.

Experimental setup

Datasets: OpenWebText CodeParrot ArXiv GSM8k WikiText-2

Benchmarks: OpenWebText CodeParrot ArXiv GSM8k WikiText-2

Metrics: KL divergence flip rate perplexity recomputation rate

Baselines: uniform low-precision accumulation random recomputation GPT-2 small uniform TF32 accumulation uniform BF16 accumulation

Builds on: mixed-precision matrix multiplication FlashAttention online softmax

Closest papers by meaning

• Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference (NeurIPS 2025 · similarity 0.87)
• Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models (NeurIPS 2024 · similarity 0.86)
• Entropy-Gated Latent Recursion (NeurIPS 2026 · similarity 0.86)
• LeaPformer: Enabling Linear Transformers for Autoregressive and Simultaneous Tasks via Learned Proportions (ICML 2024 · similarity 0.86)
• Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Tasks (ICML 2024 · similarity 0.86)
• Learning from Students: Applying t-Distributions to Explore Accurate and Efficient Formats for LLMs (ICML 2024 · similarity 0.86)