Research map / Large language models / Reasoning and chain of thought

SPD: Sync-Point Drop for Efficient Tensor Parallelism of Large Language Models

ICML 2025 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Sync-Point Drop (SPD), which removes the attention-output all-reduce in tensor-parallel LLM inference using modified block designs and sensitivity-based per-block strategies (zero-shot dropping, block-to-block distillation, head-grouping initialization).

Problem

Tensor parallelism for distributed LLM inference requires an all-reduce sync-point after every block, and these communication costs become a latency bottleneck, especially on low-bandwidth interconnects.

Approach

Drop the all-reduce after the self-attention output and restructure the residual connections in the MLP input and output so that information loss is minimized; then assign SPD to blocks according to measured perplexity sensitivity, using zero-shot dropping for insensitive blocks, block-to-block distillation for sensitive blocks, and attention-head grouping initialization plus distillation for extr

Key results

• On LLaMA2-70B with low-bandwidth interconnect, 70% SPD gives about 19.7% speedup with only 0.94% accuracy loss on zero-shot tasks.
• Zero-shot dropping alone can drop at least 44% of blocks with little accuracy loss across models; for LLaMA2-70B 8-GPU, 75% of blocks are insensitive.
• Block-to-block distillation recovers up to +28% accuracy on LLaMA2-13B 8-GPU at 100% SPD compared with zero-shot dropping.

Measured effects reported in the paper

Surprising findings

• Dropping the MLP-input/residual sync for OPT models shows less degradation than LLaMA2, attributed to higher redundancy in OPT.
• Placing the attention-output residual before the MLP all-reduce while placing the bias residual after it is required; wrong placement inflates perplexity sharply (e.g., 5.47 to 177.69 for LLaMA2-7B).

Limitations

• Accuracy degradation is larger for LLaMA2 than OPT, and extremely sensitive blocks appear only in smaller models.
• Gains depend on interconnect bandwidth; high-bandwidth setups show smaller latency benefits.
• Evaluation focuses on TTFT (prefill) latency and zero-shot/MMLU accuracy; decoding behaviour is less explored in the text.

Experimental setup

Datasets: WikiText2

Benchmarks: ARC HellaSwag LAMBADA PIQA SciQ WinoGrande MMLU WikiText2 perplexity

Metrics: average zero-shot accuracy MMLU accuracy perplexity time to first token speedup all-reduce data transfer latency

Baselines: No SPD (standard tensor parallelism) zero-shot dropping (ZS)

Builds on: tensor parallelism Megatron-LM GPTQ

Closest papers by meaning

• Tensor-Parallelism with Partially Synchronized Activations (NeurIPS 2025 · similarity 0.92)
• Synergistic Tensor and Pipeline Parallelism (NeurIPS 2025 · similarity 0.89)
• SpeedLoader: An I/O efficient scheme for heterogeneous and distributed LLM operation (NeurIPS 2024 · similarity 0.88)
• Ladder-Residual: Parallelism-Aware Architecture for Accelerating Large Model Inference with Communication Overlapping (ICML 2025 · similarity 0.88)
• Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsity (NeurIPS 2025 · similarity 0.87)
• Cost-Efficient LLM Training with Lifetime-Aware Tensor Offloading via GPUDirect Storage (NeurIPS 2025 · similarity 0.87)