Research map / Large language models / Reasoning and chain of thought

The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that the inference backend alone (vLLM, SGLang, llama.cpp, LMDeploy, Ollama vs. transformers) shifts LLM benchmark scores by up to 16.6 points under fixed weights and decoding, traces causes to engine defaults and numerical drift, and surveys 35k ML papers finding the inference stack is rarely reported.

Problem

LLM benchmark results are typically assumed to depend only on model weights and evaluation protocol, but inference engines with system-level optimizations may alter token probabilities and generations, making benchmark comparisons irreproducible.

Approach

Holds weights, prompts, decoding (greedy), precision and hardware fixed while swapping inference engines, measures accuracy, disagreement, length, divergence index and logprob drift, and isolates causes via targeted ablations; complemented by a landscape survey of 200 engines and an LLM-judged survey of publications on inference-stack reporting.

Key results

• Backend choice shifts DeepSeek R1 Distill Qwen 7B GSM8K accuracy by 16.60 points between best and worst backends (Ollama 61.87 vs. transformers 78.47).
• Disabling Ollama's hidden default repetition penalty (1.1) raises DeepSeek R1 GSM8K accuracy by 11.67 points; removing a hidden BOS token recovers 8.34 points for Llama 3.1 on Ollama.
• Of 35,000+ ML papers, the survey identified 9,018 relevant papers; among 2,257 papers without code, 36% named the inference backend.

Surprising findings

• Ollama's hidden default repetition penalty of 1.1 substantially degrades chain-of-thought accuracy for reasoning models.
• Disabling prefix caching or CUDA graphs causes random accuracy shifts in both directions, with no single failing layer identified.

Limitations

• Controlled setup (batch size 1, greedy decoding, fixed FP16) may not reflect production configurations, so variance may be a lower bound.
• Greedy decoding may be suboptimal for reasoning models, potentially skewing their scores.
• Ablating every custom kernel is infeasible; full determinism is often impossible without abandoning the engine.

Experimental setup

Datasets: GSM8K GPQA Diamond SimpleQA Verified LiveCodeBench v6 JailbreakBench

Benchmarks: GSM8K GPQA Diamond SimpleQA Verified LiveCodeBench v6 JailbreakBench

Metrics: accuracy F1 pass@1 attack success rate disagreement rate length error normalized divergence score logprob RMSE top-5 Jaccard similarity

Baselines: transformers vLLM SGLang llama.cpp LMDeploy Ollama

Builds on: vLLM SGLang llama.cpp

Closest papers by meaning

• Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference (NeurIPS 2025 · similarity 0.93)
• TokenPowerBench: Benchmarking the Power Consumption of LLM Inference (AAAI 2026 · similarity 0.91)
• DISC: Dynamic Decomposition Improves LLM Inference Scaling (NeurIPS 2025 · similarity 0.91)
• Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs (ICLR 2026 · similarity 0.90)
• xLSTM 7B: A Recurrent LLM for Fast and Efficient Inference (ICML 2025 · similarity 0.90)
• Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for LLM Problem-Solving (ICLR 2025 · similarity 0.90)