Research map / Architectures and efficiency / RNNs and normalization

ClusterFusion: Expanding Operator Fusion Scope for LLM Inference via Cluster-Level Collective Primitive

NeurIPS 2025 · Architectures and efficiency › RNNs and normalization · approach: Distributed LLM inference systems · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ClusterReduce and ClusterGather primitives over Hopper distributed shared memory and a cluster-centric dataflow that fuses QKV Projection, Attention, and Output Projection into one kernel for faster LLM decoding.

Problem

LLM decoding latency is high because operators run as fragmented kernels that exchange and reduce intermediate data through off-chip global memory, and Hopper's distributed shared memory (DSMEM) lacks structured collective communication abstractions.

Approach

Treat each thread block cluster as a cooperative unit and provide ClusterReduce and ClusterGather collectives over DSMEM (binary-tree exchange) so that QKV Projection, Attention and Output Projection can be fused into a single kernel with intermediates kept on-chip.

Key results

• On Llama2-7B with cluster size 4 at batch size 1, ClusterFusion achieves average TPOT speedups of 1.41x over SGLang, 1.39x over vLLM, 1.43x over TensorRT-LLM and 2.03x over MLC-LLM.
• For core modules on Llama2-7B, average speedups are 1.85x, 1.73x, 1.61x and 3.19x over SGLang, vLLM, TensorRT-LLM and MLC-LLM respectively.
• Disabling DSMEM increases TPOT by up to 33%; at batch size 16 the average end-to-end speedups shrink to about 1.07x-1.32x.

Measured effects reported in the paper

Surprising findings

• At 16 TBs per cluster, DSMEM bandwidth (2.90 TB/s) slightly lags global memory bandwidth (2.96 TB/s), so larger clusters can be counterproductive.
• The SplitHead dataflow, which keeps intermediates in registers, yields higher latency than SplitToken because its DSMEM traffic grows with sequence length.

Limitations

• Fusion scope is bounded by cluster size (max 16 thread blocks on Hopper); larger hidden dimensions may exceed it and fall back to global memory.
• Speedups shrink substantially at batch size 16 because compute intensity and weight/KV traffic dominate.
• Evaluated only on H100 GPUs with two models (Llama2-7B and DeepSeek-V2-Lite).

Experimental setup

Datasets: ShareGPT Splitwise

Benchmarks: TPOT on Llama2-7B and DeepSeek-V2-Lite core module latency (QKV Projection, Attention, Output Projection)

Metrics: time per output token (TPOT) latency speedup global memory transfer size kernel launch overhead

Baselines: SGLang vLLM TensorRT-LLM MLC-LLM FlashInfer CUTLASS

Builds on: FlashDecoding FlashAttention FlashMLA weight absorption

Closest papers by meaning

• SpecExec: Massively Parallel Speculative Decoding For Interactive LLM Inference on Consumer Devices (NeurIPS 2024 · similarity 0.89)
• Break the Sequential Dependency of LLM Inference Using Lookahead Decoding (ICML 2024 · similarity 0.89)
• Speculate Deep and Accurate: Lossless and Training-Free Acceleration for Offloaded LLMs via Substitute Speculative Decoding (NeurIPS 2025 · similarity 0.89)
• SpeedLoader: An I/O efficient scheme for heterogeneous and distributed LLM operation (NeurIPS 2024 · similarity 0.89)
• Practical Offloading for Fine-Tuning LLM on Commodity GPU via Learned Sparse Projectors (AAAI 2025 · similarity 0.88)
• EasySpec: Layer-Parallel Speculative Decoding for Efficient Multi-GPU Utilization (NeurIPS 2025 · similarity 0.88)