Research map / Optimization / Distributed and compressed

Flag Aggregator: Scalable Distributed Training under Failures and Augmented Losses using Convex Optimization

ICLR 2024 · Optimization › Distributed and compressed · approach: Local and asynchronous SGD · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Flag Aggregator, a subspace-based robust gradient aggregation for distributed SGD that formulates aggregation as a maximum-likelihood problem over Grassmannian/flag subspaces and solves it with IRLS using repeated weighted SVDs.

Problem

Distributed training of deep models is vulnerable to Byzantine workers (hardware/software failures, augmentation issues, adversarial or noisy gradients), and existing robust aggregators are heuristic and lack principled optimization formulations and convergence guarantees.

Approach

Aggregate worker gradients by finding an orthonormal subspace that best reconstructs the gradient matrix under a beta-distributed worker-value model, solved via iteratively reweighted least squares (IRLS) whose inner steps are weighted SVDs; the resulting YY^T projection acts like a PSD preconditioner, giving convergence whenever plain SGD converges.

Key results

• Flag Aggregator achieves higher accuracy than baseline aggregators under random-vector Byzantine attacks with p=15 workers and f=2 faulty workers on image classification.
• Rebuttal reports additional results against Fall of Empires and 10x sign-flip attacks, with FA remaining robust.
• Per-iteration cost is dominated by the server-side IRLS/SVD step, so wall-clock gains over baselines are modest.

Surprising findings

• Uniform-weight single FA iteration reduces to PCA, so the reweighting is the key novelty over PCA-based aggregation.

Limitations

• Experiments limited to small models (ResNet18, 2-layer CNN) and image classification; no language model evaluation.
• Server-side SVD/IRLS cost adds wall-clock overhead, making time-to-accuracy gains marginal.
• Byzantine attacks mainly simulated with uniformly random vectors in main experiments; stronger attacks only in rebuttal material.

Experimental setup

Datasets: CIFAR10 MNIST Tiny ImageNet

Benchmarks: CIFAR10 MNIST Tiny ImageNet

Metrics: test accuracy time-to-accuracy

Baselines: Bulyan PCA Krum coordinate-wise median geometric median

Builds on: Flag Median FlagIRLS PCA

Closest papers by meaning

• Fault Tolerant ML: Efficient Meta-Aggregation and Synchronous Training (ICML 2024 · similarity 0.90)
• From Inexact Gradients to Byzantine Robustness: Acceleration and Optimization under Similarity (NeurIPS 2026 · similarity 0.88)
• On the Tension between Byzantine Robustness and No-Attack Accuracy in Distributed Learning (ICML 2025 · similarity 0.88)
• Byzantine-Robust Decentralized Learning via Remove-then-Clip Aggregation (AAAI 2024 · similarity 0.88)
• Weight for Robustness: A Comprehensive Approach towards Optimal Fault-Tolerant Asynchronous ML (NeurIPS 2024 · similarity 0.87)
• Robust Distributed Gradient Aggregation Using Projections onto Gradient Manifolds (AAAI 2024 · similarity 0.87)