Research map / Large language models / Reasoning and chain of thought
Tensor-Parallelism with Partially Synchronized Activations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CAAT-Net, a tensor-parallel architecture that synchronizes only a fraction of activation channels (partial channel-reduce), cutting communication up to 50% with little accuracy loss, and speeds up training and inference.
Problem
Tensor-parallel LLM training and inference are bottlenecked by all-reduce communication of activations, and the cost grows as the tensor-parallel degree increases relative to compute.
Approach
Replace the full all-reduce with a partial channel-reduce that only sums a fixed fraction p of hidden channels (shared channels), keeping the rest private per device; the model is trained from scratch with this architecture, using backward-pass all-reduce moved after normalization, fp32 gradient accumulation, and private-channel variance scaling.
Key results
- 7B CAAT-Net (p=0.5, TP=8) trained on 160B tokens matches baseline on nearly all zero-shot benchmarks, with validation loss 1.00 vs 1.01 for the baseline, while halving tensor-parallel communication.
- Training throughput at TP=16 improves up to 14% with p=0.25 and 9% with p=0.5 on Gaudi3.
- Inference TTFT speedup is up to 26% at TP=16 with p=0.25 (batch 32) and up to 13% on NVIDIA H100/A100 with p=0.25 at TP=8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use partial channel–reduce with p=0.5 instead of full synchronization | Llama2-7B · Zero-shot benchmarks | accuracy: LAMBADA 61.34 ± 0.68 -> 61.05 ± 0.68; HellaSwag 45.85 ± 0.50 -> 46.10 ± 0.50; WinoGrande 61.48 ± 1.37 -> 62.19 ± 1.36; P | Mixed |
| Use partial channel–reduce with p=0.5 instead of full synchronization | Llama2-7B · WikiText and validation | perplexity and validation loss: WikiText 12.51 -> 12.46; Validation Loss 1.01 -> 1.00 | Helped |
| Reduce synchronization factor from p=1 to p=0.75 (ablation) | TinyLlama · Validation | loss: 1.28 -> 1.27 | Helped |
| Reduce synchronization factor below p=0.5 (ablation) | TinyLlama · Validation | loss: 1.28 -> 1.29 (p=0.375); 1.28 -> 1.29 (p=0.25); 1.28 -> 1.31 (p=0.125); 1.28 -> 1.49 (p=0) | Hurt |
| Add private channel scaling (ablation) | CAAT-Net · 130M model, tensor-parallel 8, p=0.5 | validation loss | Helped |
| Accumulate backward-pass all-reduce gradients in 32-bit rather than 16-bit (ablation) | TinyLlama training · Training loss | loss | Helped |
| Replace partial channel–reduce with Top-K or random activation masking (ablation) | 130M Llama model · Pretraining | validation loss | Hurt |
| Reduce synchronization factor from p=1 to p=0.5 (ablation) | GPT-XL · Zero-shot benchmarks | accuracy: LAMBADA 50.16 ± 0.70 -> 48.17± 0.69; HellaSwag 36.07 ± 0.48 -> 36.26 ± 0.48; WinoGrande 52.41 ± 1.40 -> 53.83 ± 1.40; PI | Mixed |
Surprising findings
- Reducing communication can slightly improve validation loss at intermediate p values (e.g., 130M and 1.1B models).
- Top-K and random activation masking cause severe degradation even at minimal compression, unlike CAAT-Net.
Limitations
- Training speedup experiments used 16-bit backward accumulation due to implementation limits, so training throughput results are a projection.
- Most accuracy differences are not statistically significant; BoolQ shows a marginal drop for the 7B model (62.51 vs 64.89).
Experimental setup
Datasets: RedPajama
Benchmarks: LAMBADA HellaSwag WinoGrande PIQA OpenBookQA BoolQ WikiText-103
Metrics: zero-shot accuracy perplexity validation loss tokens per second time-to-first-token
Baselines: Megatron-LM baseline (full all-reduce) Top-K activation masking random activation masking Llama2-7B baseline Optimum-Habana
Builds on: Megatron-LM Llama 2 TinyLlama Domino Ladder Residual
Closest papers by meaning
- SPD: Sync-Point Drop for Efficient Tensor Parallelism of Large Language Models (ICML 2025 · similarity 0.92)
- Synergistic Tensor and Pipeline Parallelism (NeurIPS 2025 · similarity 0.91)
- Rethinking Memory and Communication Costs for Efficient Data Parallel Training of Large Language Models (NeurIPS 2024 · similarity 0.87)
- Ladder-Residual: Parallelism-Aware Architecture for Accelerating Large Model Inference with Communication Overlapping (ICML 2025 · similarity 0.87)
- SpeedLoader: An I/O efficient scheme for heterogeneous and distributed LLM operation (NeurIPS 2024 · similarity 0.86)
- EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.