Research map / Architectures and efficiency / RNNs and normalization
Speeding up Log-Sum-Exp: Kernel Fusion at the Memory Wall, Integer Arithmetic at the Compute Wall
Lingyun Yao, Martin Andraud, Niki Loppi, Andrea Pilzer, Anji Liu, Guy Van den Broeck, Martin Trapp
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes faster Log-Sum-Exp implementations on CPUs and GPUs that fuse kernels to address memory-bound execution and use integer arithmetic to address compute-bound execution.
Problem
Log-Sum-Exp (LSE) is a core numerical primitive in ML, but its efficient computation on generic CPUs and GPUs is underexplored; the de facto torch.logsumexp GPU implementation launches nine sub-kernels per call with redundant memory traffic.
Approach
Fuse the chain of LSE sub-kernels into a single kernel to cut memory traffic (memory wall), and replace costly floating-point exponential/log computation with integer arithmetic to relieve the compute wall.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fuse the nine torch.logsumexp sub-kernels into one single-pass kernel (FuseLSE) | GPU LSE · Microbenchmark, V100, B=512 | speedup over torch.logsumexp: ~10-11x for N<=1024; ~6x for N>=8192 | Improved |
| replace exp/log with integer approximations (IntLSE) instead of FuseLSE on GPU (ablation) | FuseLSE on GPU · Microbenchmark, V100 | latency: 6.24x vs 6.18x | No clear effect |
| replace exp/log with integer arithmetic (IntLSE) instead of FuseLSE on CPU (ablation) | FuseLSE on CPU · CPU microbenchmark, Xeon Gold 6134 | speed: 3.1-6.1x faster | Improved |
| use IntLSE on CPU instead of torch.logsumexp for narrow-range inputs U(-1,0) | torch.logsumexp on CPU · CPU microbenchmark U(-1,0), N=128,256 | speed: 1.54x slower | Hurt |
| add the 64-entry correction LUT to IntLSE (ablation) | IntLSE · Per-step pair error | maximum pointwise error: 0.085 -> 0.0068 | Improved |
| add the correction LUT to IntLSE (ablation) | IntLSE · Microbenchmark wider ranges, large N | absolute error: ~20-80x lower error on wider ranges | Improved |
| store inputs in FP16 (FuseLSE16) instead of FP32 (ablation) | FuseLSE · Microbenchmark V100, N=128,256 | speedup over torch: 6.25x -> 12.06x | Improved |
| store inputs as INT16 Q10.6 (IntLSE-Q10.6) instead of Q16.16 INT32 (ablation) | IntLSE · Microbenchmark V100, N=128,256 | mean absolute error: error ~1% at N=128,256, four orders of magnitude above FuseLSE16 | Hurt |
| use FuseLSE/IntLSE in TRL selective_log_softmax | TRL SLS · RLHF SLS, four vocabulary sizes, V100 | speedup over torch.logsumexp: 5.96-6.30x | Improved |
| use FuseLSE in LM-head + SLS stage | GPT-2 LM-head + SLS · V100, B=8, T=1024, V=50K | stage time: 59.12 -> 49.20 ms | Improved |
| use FuseLSE in sparse PyJuice sum layers | PyJuice default · HCLT on MNIST, V100 | forward-pass time: ~1.7x speedup | Improved |
| use IntLSE in PyJuice circuits instead of FuseLSE (ablation) | FuseLSE · HCLT on MNIST | log-likelihood error: 3.0-4.2e-4 -> 0.45-1.26 nats | Hurt |
Experimental setup
Metrics: runtime speedup
Baselines: torch.logsumexp
Builds on: torch.logsumexp
Closest papers by meaning
- SpeedLoader: An I/O efficient scheme for heterogeneous and distributed LLM operation (NeurIPS 2024 · similarity 0.84)
- KernelBench: Can LLMs Write Efficient GPU Kernels? (ICML 2025 · similarity 0.83)
- FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision (NeurIPS 2024 · similarity 0.83)
- FlashMoE: Fast Distributed MoE in a Single Kernel (NeurIPS 2025 · similarity 0.83)
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (ICLR 2024 · similarity 0.82)
- FloE: On-the-Fly MoE Inference on Memory-constrained GPU (ICML 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
