Research map / Optimization / Distributed and compressed
DeToNATION: Decoupled Torch Network-Aware Training on Interlinked Online Nodes
Mogens Henrik From, Jacob Nielsen, Lukas Galke, Peter Schneider-Kamp
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace DeMo DCT replication with Random replication (same compression) | FlexDeMo-SGD · T5 Opus Books Fr-En | validation loss | Improved |
| replace Random with DeMo replication | FlexDeMo-SGD · ViT-B CIFAR-100 | validation loss | Improved |
| replace Random with DeMo replication | FlexDeMo-SGD · OLMo2 1B Dolma v1.6 | train loss | Improved |
| use Striding replication instead of DeMo | FlexDeMo-SGD · OLMo2 1B Dolma v1.6 | train loss | Hurt |
| use DiLoCo replication instead of DeMo | FlexDeMo-SGD · T5 Opus Books Fr-En | validation loss | Hurt |
| share only the sign of the gradient components (ablation) | DeMo replication · T5, ViT, OLMo hyperparameter study (Appendix) | loss convergence | Improved |
| use fp32 instead of fp16 transfer dtype (ablation) | DeMo replication · Hyperparameter study (Appendix) | loss convergence | Improved |
| increase chunk size / TopK (ablation) | DeMo replication · Hyperparameter study (Appendix) | loss convergence | Mixed |
| use FlexDeMo instead of Hybrid-FSDP with full-gradient AdamW | Hybrid-FSDP · OLMo2 1B Dolma v1.6, 2 nodes | wall-clock time: about 2.6 times faster | Improved |
| use FlexDeMo instead of Hybrid-FSDP with full-gradient AdamW | Hybrid-FSDP · OLMo2 1B Dolma v1.6, 2 nodes | train loss | Improved |
| replace Random with DeMo replication at 64 nodes | FlexDeMo-SGD · OLMo2 1B, 64 nodes | wall-clock time | Hurt |
| use Random instead of DeMo replication at 64 nodes | FlexDeMo-SGD · OLMo2 1B, 64 nodes | train loss: 64% faster than conventional setup | Hurt |
Closest papers by meaning
- Towards a Better Theoretical Understanding of Independent Subnetwork Training (ICML 2024 · similarity 0.85)
- Synergistic Tensor and Pipeline Parallelism (NeurIPS 2025 · similarity 0.85)
- CO2: Efficient Distributed Training with Full Communication-Computation Overlap (ICLR 2024 · similarity 0.85)
- Harnessing Manycore Processors with Distributed Memory for Accelerated Training of Sparse and Recurrent Models (AAAI 2024 · similarity 0.84)
- Unextractable Protocol Models: Collaborative Training and Inference without Weight Materialization (NeurIPS 2025 · similarity 0.84)
- MARS: Unleashing the Power of Variance Reduction for Training Large Models (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
