Research map / Federated learning and privacy / Communication-efficient FL
Interlocking Backpropagation: Improving depthwise model-parallelism
Aidan N. Gomez, Oscar Key, Kuba Perlin, Stephen Gou, Nick Frosst, Jeff Dean, Yarin Gal
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add 2-wise interlocking to local (1-wise) training (ablation) | ResNet-32 · CIFAR-100 test | accuracy (%): 75.02 -> 78.09 | Improved |
| add 2-wise interlocking to local (1-wise) training (ablation) | ResNet-50 · ImageNet val | accuracy (%): 72.05 -> 74.45 | Improved |
| increase interlocking from 2-wise to 3-wise (ablation) | ResNet-50 · ImageNet val | accuracy (%): 74.45 -> 76.27 | Improved |
| increase interlocking from 2-wise to 3-wise (ablation) | ResNet-32 · CIFAR-100 test | accuracy (%): 78.09 -> 77.84 | No clear effect |
| replace end-to-end training with 2-wise interlocking | ResNet-32 · CIFAR-100 test | accuracy (%): 76.71 -> 78.09 | Improved |
| replace end-to-end training with 3-wise interlocking | ResNet-32 · CIFAR-10 test | accuracy (%): 95.20 -> 95.42 | Improved |
| use 2-wise interlocking instead of end-to-end under fixed time budget | ResNet-32 · CIFAR-100 test (fixed time budget) | accuracy (%): 73.98 -> 76.85 | Improved |
| use 2-wise interlocking instead of local 1-wise under fixed time budget | ResNet-32 · CIFAR-100 test (fixed time budget) | accuracy (%): 74.33 -> 76.85 | Improved |
| use fine-tuning instead of 2-wise interlocking under fixed time budget | ResNet-32 · CIFAR-10 test (fixed time budget) | accuracy (%): 94.06 -> 94.21 | No clear effect |
| ensemble predictions of auxiliary heads across modules (ablation) | interlocking backprop · CIFAR-10 / ensembling | accuracy | No clear effect |
| use 2-wise interlocking instead of end-to-end, fixed number of steps | Transformer · One Billion Word (4 modules, fixed steps) | test perplexity | Hurt |
| use 2-wise interlocking instead of end-to-end, per-time comparison | Transformer · One Billion Word (4 modules, 8 days) | test perplexity: less than half the training step time for similar test perplexity | Improved |
Closest papers by meaning
- Backpropagation-Free Deep Learning with Recursive Local Representation Alignment (AAAI 2023 · similarity 0.87)
- Towards Interpretable Deep Local Learning with Successive Gradient Reconciliation (ICML 2024 · similarity 0.87)
- Towards a Better Theoretical Understanding of Independent Subnetwork Training (ICML 2024 · similarity 0.86)
- Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagation (ICML 2024 · similarity 0.85)
- The Price of Locality: Why Forward-Forward Underperforms Backpropagation? (NeurIPS 2026 · similarity 0.85)
- Accelerated training through iterative gradient propagation along the residual path (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
