Research map / Optimization / Distributed and compressed
Distributed Parallel Gradient Stacking(DPGS): Solving Whole Slide Image Stacking Challenge in Multi-Instance Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DPGS, which parallelizes MIL bag-level gradient computation across submodels and stacks gradients to emulate batch stacking of variable-length WSI bags, plus DMGC, a joint gradient/weight sparsification scheme for communication-efficient distributed training.
Problem
Whole slide image MIL bags have variable instance counts, so they cannot be stacked into batches, causing slow sequential training and biased gradient estimates.
Approach
Each sub-process computes gradients for one variable-length bag and a master averages them to emulate mini-batch stacking without padding or sampling; DMGC extends DGC by compressing gradients with residual accumulation and transmitting sparse weight deltas instead of full weights.
Key results
- DPGS+DMGC improves ABMIL convergence on TCGA-Lung from 636.3s to 20.0s (about 31.8x) with accuracy rising from 92.38% to 93.90%.
- DMGC reaches 117.5x compression ratio at 0.01% keep rate on C16-Multiscale while keeping about 93.8% accuracy.
- On C16-Multiscale, MeanMIL accuracy improves from 61.71% to 71.33% with DPGS+DMGC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply DPGS+DMGC instead of classic sequential training | ABMIL · TCGA-Lung Multiscale | accuracy: 92.38% to 93.90% | Helped |
| Apply DPGS+DMGC instead of classic sequential training | ABMIL · TCGA-Lung Multiscale | time (s): 636.3s to 20.0s | Helped |
| Apply DPGS+DMGC instead of classic sequential training | MeanMIL · C16-Multiscale | accuracy: 61.71% to 71.33% | Helped |
| Add DMGC to DPGS (ablation) | ABMIL · C16-Multiscale | time (s): 544 ±37 to 142 ±11 | Helped |
| Add DMGC to DPGS (ablation) | ABMIL · C16-Multiscale | accuracy: 93.84% ±0.08 to 93.83% ±0.09 | No clear effect |
| Replace DGC with DMGC (ablation) | ABMIL · C16-Multiscale | time (s): 249 ±15 to 142 ±11 | Helped |
| Add DMGC to DPGS (ablation) | TransMIL · C16-Multiscale | time (s): 1311 ±90 to 313 ±19 | Helped |
| Replace DGC with DMGC (ablation) | TransMIL · C16-Multiscale | time (s): 652 ±55 to 313 ±19 | Helped |
| Vary DMGC keep rate across reported values (ablation) | ABMIL · C16-Multiscale, 10Mbps | accuracy: 93.8±0.2 to 94.0±0.1 | No clear effect |
| Set DMGC keep rate to 0.01% instead of 100% (ablation) | ABMIL · C16-Multiscale, 10Mbps | communication size (×1e3): 979.7±17.5 to 8.3±0.4 | Helped |
| Use sampling instead of DPGS stacking | MIL models · Camelyon16 datasets | accuracy | Hurt |
Surprising findings
- Sampling accelerates convergence but reduces accuracy and increases variance, and padding is slower than classic training on some datasets.
- Keeping only 0.01% of gradients still preserves accuracy (about 93.8%) while reducing communication, with non-monotonic convergence time vs. discard rate.
Limitations
- Synchronization overhead still limits GPU utilization, as the authors note.
- Speedup and accuracy gains vary by model and dataset; some models (e.g., MeanMIL on C16-ImageNet) did not converge under classic training.
- Convergence time is defined by a loss-percentile criterion, and comparisons rely on single-GPU baselines versus multi-node DPGS.
Experimental setup
Datasets: Camelyon16 TCGA-Lung
Benchmarks: Camelyon16 TCGA-Lung
Metrics: accuracy convergence time communication size compression ratio
Baselines: ABMIL MeanMIL TransMIL CLAM-MB ACMIL RRTMIL Padding Sampling DGC
Builds on: Deep Gradient Compression CLAM TransMIL ABMIL Goyal et al. linear scaling rule
Closest papers by meaning
- Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training (ICLR 2018 · similarity 0.85)
- AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training (AAAI 2018 · similarity 0.84)
- Leveraging partial stragglers within gradient coding (NeurIPS 2024 · similarity 0.82)
- Sketched Adaptive Distributed Deep Learning: A Sharp Convergence Analysis (NeurIPS 2025 · similarity 0.82)
- Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism (NeurIPS 2025 · similarity 0.82)
- Approximate Gradient Coding for Distributed Learning with Heterogeneous Stragglers (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.