Research map / Large language models / Reasoning and chain of thought
Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
The paper shows that BF16 inference makes greedy decoding and sampling-based benchmark results of reasoning LLMs vary across GPU count, GPU type, and batch size, and proposes LayerCast (BF16 storage, FP32 compute) as a cheaper route to near-FP32 reproducibility.
Problem
LLM benchmark results under greedy decoding and random sampling are not reproducible across hardware and system configurations (GPU count, GPU type, evaluation batch size) because of floating-point rounding under limited precision, especially for long reasoning chains.
Approach
Systematically quantify how BF16, FP16 and FP32 precision drive divergence in greedy outputs and variance in sampled Pass@1 across 12 runtime configurations, and propose LayerCast, which stores weights in 16-bit but upcasts each linear layer's weights to FP32 just-in-time so all computation is FP32.
Key results
- Under BF16 greedy decoding, DeepSeek-R1-Distill-Qwen-7B shows 9.15% Std@Acc on AIME'24 across 12 configurations, versus 0 under FP32.
- With FP32, only 2.2% of MATH500 samples diverge for DeepSeek-R1-Distill-Qwen-7B, versus over 90% under BF16.
- LayerCast reaches stability nearly identical to FP32 with divergence rates below 3.4% and 34% lower memory than full FP32.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FP32 instead of BF16 | DeepSeek-R1-Distill-Qwen-7B greedy inference · AIME'24 across 12 runtime configurations | Std@Acc: 9.15% -> 0 | Helped |
| Use FP32 instead of BF16 | DeepSeek-R1-Distill-Qwen-7B greedy inference · MATH500 across 12 runtime configurations | Avg_Std@Output_Length: 2774.28 -> 138.75 | Helped |
| Use FP16 instead of BF16 | DeepSeek-R1-Distill-Qwen-7B greedy inference · AIME'24 across 12 runtime configurations | Std@Acc: 9.15% -> 5.74% | Helped |
| Use FP32 instead of BF16 | DeepSeek-R1-Distill-Qwen-7B random-sampling inference · MATH500 (n=4) | Std of Pass@1: 0.3158 -> 0.1021 | Helped |
| Use FP32 instead of BF16 | DeepSeek-R1-Distill-Qwen-7B random-sampling inference · AIME'24 (n=64) | Std of Pass@1: 0.3749 -> 0.7377 | Hurt |
| Store weights in BF16 and perform computations in FP32 (LayerCast) | DeepSeek-R1-Distill-Qwen-7B inference · Greedy decoding across runtime configurations | Output divergence: below 3.4% | Helped |
| Store weights in BF16 and perform computations in FP32 (LayerCast) | LLM inference · Inference memory use | Memory usage reduction vs full FP32: 34% | Helped |
| Use FP32 instead of BF16 | Qwen3-32B greedy inference · AIME'24 across runtime configurations | Std@Acc: 5.02% -> 1.11% | Helped |
| Use FP32 instead of BF16 | DeepSeek-R1-Distill-Llama-8B greedy inference · GPQA Diamond across runtime configurations | Std@Acc: 3.03% -> 1.96% | Helped |
| Increase GPU count from 2 to 4 (ablation) | LLM greedy inference on L40S · Across tested benchmarks and models | Avg_Std@top1_prob | Mixed |
| Reduce batch size (ablation) | LLM greedy inference · Across tested runtime configurations | Avg_Std@top1_prob | Hurt |
| Use A100 instead of L40S (ablation) | LLM greedy inference · Matched runtime configurations | Avg_Std@top1_prob | Hurt |
Surprising findings
- Smaller batch sizes tend to produce higher top-1 probability variance than larger batches.
- Greedy decoding with BF16 can change response length by up to 9,000 tokens across system configurations.
Limitations
- Experiments limited to A100 and L40S GPUs and a few models and tasks
- Mechanistic explanations (e.g., batch-size and GPU-count effects) are partly speculative
- LayerCast's memory saving is only 34% and it is a vLLM patch
Experimental setup
Datasets: AIME'24 MATH500 LiveCodeBench-Easy LiveCodeBench-Medium LiveCodeBench-Hard GPQA Diamond
Benchmarks: AIME'24 MATH500 LiveCodeBench-Easy LiveCodeBench-Medium LiveCodeBench-Hard GPQA Diamond
Metrics: Std@Acc Avg_Std@Output_Length Div_Index Div_Percent Avg_Std@top1_prob Pass@1
Baselines: BF16 inference FP16 inference FP32 inference
Builds on: vLLM SkyThought Batch-invariant operations
Closest papers by meaning
- The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility (NeurIPS 2026 · similarity 0.93)
- Learning from Students: Applying t-Distributions to Explore Accurate and Efficient Formats for LLMs (ICML 2024 · similarity 0.90)
- TokenPowerBench: Benchmarking the Power Consumption of LLM Inference (AAAI 2026 · similarity 0.90)
- Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens (ICML 2026 · similarity 0.90)
- Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for LLM Problem-Solving (ICLR 2025 · similarity 0.90)
- Bag of Tricks for Inference-time Computation of LLM Reasoning (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.