Research map / Architectures and efficiency / RNNs and normalization

FRESCO: A Novel Consistency Control for Asynchronous Pipeline Parallel Training

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Distributed LLM inference systems

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FRESCO, a consistency control for asynchronous pipeline parallel training that addresses gradient inconsistency by exploiting that error signals depend on forward-time features, not only parameter states.

Problem

Asynchronous pipeline parallelism speeds up distributed training but becomes unstable because gradient computations are inconsistent across stages.

Approach

Identifies that error signals are coupled to forward-time features rather than just parameter states, so parameters may evolve as long as they preserve forward-equivalence; FRESCO leverages this to control consistency in asynchronous pipelines.

Closest papers by meaning

• AsyncMesh: Fully Asynchronous Optimization for Data and Pipeline Parallelism (NeurIPS 2026 · similarity 0.92)
• Nesterov Method for Asynchronous Pipeline Parallel Optimization (ICML 2025 · similarity 0.89)
• Communication-Efficient LLM Adaptation over Decentralized GPU Meshes (NeurIPS 2026 · similarity 0.87)
• Beyond the Trial-and-Error Loop: Hybrid Projection and Automated Tuning for Distributed Training (NeurIPS 2026 · similarity 0.86)
• Synergistic Tensor and Pipeline Parallelism (NeurIPS 2025 · similarity 0.86)
• PipeOffload: Improving Scalability of Pipeline Parallelism with Memory Optimization (ICML 2025 · similarity 0.86)