AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

Speeding up Log-Sum-Exp: Kernel Fusion at the Memory Wall, Integer Arithmetic at the Compute Wall

Lingyun Yao, Martin Andraud, Niki Loppi, Andrea Pilzer, Anji Liu, Guy Van den Broeck, Martin Trapp

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Distributed LLM inference systems · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes faster Log-Sum-Exp implementations on CPUs and GPUs that fuse kernels to address memory-bound execution and use integer arithmetic to address compute-bound execution.

Problem

Log-Sum-Exp (LSE) is a core numerical primitive in ML, but its efficient computation on generic CPUs and GPUs is underexplored; the de facto torch.logsumexp GPU implementation launches nine sub-kernels per call with redundant memory traffic.

Approach

Fuse the chain of LSE sub-kernels into a single kernel to cut memory traffic (memory wall), and replace costly floating-point exponential/log computation with integer arithmetic to relieve the compute wall.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
fuse the nine torch.logsumexp sub-kernels into one single-pass kernel (FuseLSE)GPU LSE · Microbenchmark, V100, B=512speedup over torch.logsumexp: ~10-11x for N<=1024; ~6x for N>=8192Improved
replace exp/log with integer approximations (IntLSE) instead of FuseLSE on GPU (ablation)FuseLSE on GPU · Microbenchmark, V100latency: 6.24x vs 6.18xNo clear effect
replace exp/log with integer arithmetic (IntLSE) instead of FuseLSE on CPU (ablation)FuseLSE on CPU · CPU microbenchmark, Xeon Gold 6134speed: 3.1-6.1x fasterImproved
use IntLSE on CPU instead of torch.logsumexp for narrow-range inputs U(-1,0)torch.logsumexp on CPU · CPU microbenchmark U(-1,0), N=128,256speed: 1.54x slowerHurt
add the 64-entry correction LUT to IntLSE (ablation)IntLSE · Per-step pair errormaximum pointwise error: 0.085 -> 0.0068Improved
add the correction LUT to IntLSE (ablation)IntLSE · Microbenchmark wider ranges, large Nabsolute error: ~20-80x lower error on wider rangesImproved
store inputs in FP16 (FuseLSE16) instead of FP32 (ablation)FuseLSE · Microbenchmark V100, N=128,256speedup over torch: 6.25x -> 12.06xImproved
store inputs as INT16 Q10.6 (IntLSE-Q10.6) instead of Q16.16 INT32 (ablation)IntLSE · Microbenchmark V100, N=128,256mean absolute error: error ~1% at N=128,256, four orders of magnitude above FuseLSE16Hurt
use FuseLSE/IntLSE in TRL selective_log_softmaxTRL SLS · RLHF SLS, four vocabulary sizes, V100speedup over torch.logsumexp: 5.96-6.30xImproved
use FuseLSE in LM-head + SLS stageGPT-2 LM-head + SLS · V100, B=8, T=1024, V=50Kstage time: 59.12 -> 49.20 msImproved
use FuseLSE in sparse PyJuice sum layersPyJuice default · HCLT on MNIST, V100forward-pass time: ~1.7x speedupImproved
use IntLSE in PyJuice circuits instead of FuseLSE (ablation)FuseLSE · HCLT on MNISTlog-likelihood error: 3.0-4.2e-4 -> 0.45-1.26 natsHurt

Experimental setup

Metrics: runtime speedup

Baselines: torch.logsumexp

Builds on: torch.logsumexp

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy