Research map / Architectures and efficiency / RNNs and normalization
Ladder-Residual: Parallelism-Aware Architecture for Accelerating Large Model Inference with Communication Overlapping
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Ladder Residual, a residual-stream rerouting that feeds each module a stale input so tensor-parallel AllReduce can overlap with computation, yielding up to 29% inference speedup on 70B and comparable accuracy when trained from scratch or adapted from Llama-3.1 8B.
Problem
Tensor parallelism requires blocking AllReduce operations after each attention and MLP block, and this communication latency limits inference scaling across GPUs, especially with slow interconnects.
Approach
Reroute the residual stream so module i+1 consumes the input from two layers back (x_{i-1}) instead of x_i, decoupling the computation of h_{i+1} from the AllReduce of x_i and allowing asynchronous overlap, while keeping the regular residual connection so accuracy is preserved.
Key results
- Ladder Residual on a 70B Llama-style model with TP=8 gives up to 29% higher throughput with P2P enabled and up to 60% with P2P disabled.
- Hybrid-Ladder-8B-16L retrained on 3B tokens averages 57.61 vs 56.11 for Llama-3.1-8B-Instruct, with about 21% end-to-end inference speedup.
- At 3.5B scale from scratch, Ladder Transformer averages 62.91 vs 64.11 for standard transformer (about 1.2 points lower) with 14.90 vs 14.48 Wikitext perplexity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Ladder Residual instead of the standard residual architecture | Transformer inference · 1B–405B inference, with P2P enabled or disabled | throughput speedup: 1.29x–1.59x | Helped |
| Use Ladder Residual instead of the standard architecture | Llama-70B inference, P2P=1, batch size 1 · 70B inference | Token/sec Improvement (%): 30.79 | Helped |
| Use Ladder Residual instead of the standard architecture | Llama-70B inference, P2P=0, batch size 1 · 70B inference | Token/sec Improvement (%): 59.87 | Helped |
| Train with Ladder Residual rather than standard residuals | 1.2B Transformer · FineWeb-edu, 100B tokens | Average: 59.98 -> 58.92 | No clear effect |
| Train with Ladder Residual rather than standard residuals | 3.5B Transformer · FineWeb-edu, 100B tokens | Average: 64.11 -> 62.91 | Hurt |
| Apply Ladder Residual to upper 16 layers and retrain | Llama-3.1-8B-Instruct · Llama-3.1-8B-Instruct benchmark suite | Average: 56.11 -> 57.61 | Helped |
| Apply Ladder Residual to upper 16 layers without retraining (ablation) | Llama-3.1-8B-Instruct · Llama-3.1-8B-Instruct benchmark suite | Average: 56.11 -> 41.65 | Hurt |
| Retrain after applying Ladder Residual to upper 16 layers (ablation) | Hybrid-Ladder-8B-16L · Llama-3.1-8B-Instruct benchmark suite | Average: 41.65 -> 57.61 | Helped |
| Adapt 20 layers rather than the upper 16 layers, with retraining (ablation) | Llama-3.1-8B-Instruct · Llama-3.1-8B-Instruct benchmark suite | Average: 57.61 -> 53.86 | Hurt |
| Parallelize attention and MLP instead of using Ladder Residual | 1.2B Transformer · FineWeb-edu, 100B tokens | Average: 58.75 vs 58.92 | Mixed |
| Parallelize attention and MLP instead of using Ladder Residual | 3.5B Transformer · FineWeb-edu, 100B tokens | Average: 63.03 vs 62.91 | No clear effect |
Surprising findings
- Zero-shot application of Ladder Residual to Llama-3.1-8B-Instruct drops GSM8K from 84.99 to 10.54, showing the representation shift must be recovered by retraining.
- At 1.2B scale, Ladder Transformer performs similarly to the standard transformer, but at 3.5B the standard transformer is slightly better.
Limitations
- Speedups shrink with fast P2P interconnect for some model sizes (e.g., 70B P2P-enabled gains are smaller than P2P-disabled gains).
- Accuracy at 3.5B scale is slightly below the standard transformer (1.2 points average).
- Post-training adaptation requires retraining; zero-shot application severely degrades generative tasks.
Experimental setup
Datasets: FineWeb-edu Infinity-Instruct
Benchmarks: ARC-C ARC-E HellaSwag PIQA SciQ Winogrande Wikitext perplexity MMLU OpenBookQA TruthfulQA GSM8K HumanEval+
Metrics: throughput (tokens/sec) end-to-end latency speedup accuracy perplexity length-controlled win rate
Baselines: Standard Transformer Parallel Attention and MLP (PaLM) Communication-Free Upper Bound Llama-3.1-8B-Instruct
Builds on: Tensor parallelism (Megatron-style) PaLM parallel attention and MLP gpt-fast Llama-3
Closest papers by meaning
- EasySpec: Layer-Parallel Speculative Decoding for Efficient Multi-GPU Utilization (NeurIPS 2025 · similarity 0.88)
- SPD: Sync-Point Drop for Efficient Tensor Parallelism of Large Language Models (ICML 2025 · similarity 0.88)
- 70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11) (NeurIPS 2025 · similarity 0.87)
- SpeedLoader: An I/O efficient scheme for heterogeneous and distributed LLM operation (NeurIPS 2024 · similarity 0.87)
- Tensor-Parallelism with Partially Synchronized Activations (NeurIPS 2025 · similarity 0.87)
- ZeRO++: Extremely Efficient Collective Communication for Large Model Training (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.