Research map / Architectures and efficiency / RNNs and normalization
Towards a Better Theoretical Understanding of Independent Subnetwork Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives a theoretical analysis of Independent Subnetwork Training (IST) as a distributed submodel SGD using permutation sketches on quadratic models, showing linear/O(1/K) convergence in interpolation cases and an irreducible bias neighborhood in general.
Problem
Independent Subnetwork Training combines data and model parallelism to reduce communication and memory, but lacks rigorous convergence theory; prior analyses required restrictive assumptions (e.g., bounded gradient norms, Lipschitz losses, very small compression variance).
Approach
Model IST as distributed submodel gradient descent with random sketches (permutation sketches that partition parameters across clients), analyzed via matrix smoothness on quadratics; show that with preconditioning, IST matches gradient descent in the interpolation case, while in general it converges only to an irreducible neighborhood determined by a bias term.
Key results
- In the interpolation case with rescaled permutation sketches and diagonal preconditioning, IST achieves a gradient-descent-matching O(1/K) rate and, with step size 1, converges in essentially one iteration for the functional gap.
- In general heterogeneous quadratics, the expected iterate converges to x_inf = (1/n^{3/2}) sum D_i^{-1/2} b_i rather than x*, so an irreducible bias remains that cannot be removed by reducing the step size.
- On a CIFAR-10 neural network experiment with n=10 nodes, IST's loss stagnates at an error floor roughly independent of step size, unlike DGD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use simplified IST with scaled permutation sketches | Quadratic problem, heterogeneous setting · Quadratic problem, heterogeneous setting | functional error | Mixed |
| Use IST instead of DGD | Neural network trained on CIFAR-10 · CIFAR-10 neural network | training loss | Hurt |
| Increase the step size | DGD on the CIFAR-10 neural network · CIFAR-10 neural network | training loss | Helped |
| Increase the step size | IST on the CIFAR-10 neural network · CIFAR-10 neural network | time to reach the error floor | Helped |
| Use smaller step sizes γ ∈ {0.01, 0.02} | IST on the neural network problem · CIFAR-10 neural network | training loss floor | Hurt |
| Decrease γ by 10 every 1000 iterations | IST on the neural network problem · CIFAR-10 neural network | training loss | No clear effect |
| Use scaled permutation sketches in the homogeneous quadratic setting | IST · Homogeneous quadratic problem | distance to expected fixed point | Helped |
Surprising findings
- IST's error floor is essentially the same across step sizes, and decreasing the step size does not remove the neighborhood, unlike SGD.
- Reducing the step size on the neural network experiment (decaying by 10 every 1000 iterations) almost does not change performance, because the bias term grows for smaller step sizes.
Limitations
- Analysis restricted to quadratic objectives with a single local gradient step per round and exact local gradients.
- Permutation sketches with heterogeneous preconditioning are no longer unbiased; homogeneous/heterogeneous guarantees require d = qn and diagonal preconditioning assumptions.
- No partial participation, local training benefits, or straggler robustness analyzed.
Experimental setup
Datasets: CIFAR-10 ImageNet (pre-training of ResNet-50, feature extractor) synthetic Gaussian quadratic problems
Benchmarks: CIFAR-10
Metrics: log relative functional error log average loss of submodels distance to fixed point
Baselines: Distributed Gradient Descent (DGD) SGD
Builds on: Independent Subnetwork Training matrix smoothness (coordinate descent analysis) compressed gradient descent
Closest papers by meaning
- Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism (NeurIPS 2025 · similarity 0.88)
- Distributed Optimization for Overparameterized Problems: Achieving Optimal Dimension Independent Communication Complexity (NeurIPS 2022 · similarity 0.88)
- Rethinking Memory and Communication Costs for Efficient Data Parallel Training of Large Language Models (NeurIPS 2024 · similarity 0.87)
- From Promise to Practice: Realizing High-performance Decentralized Training (ICLR 2025 · similarity 0.87)
- Mixtures of Subspaces for Bandwidth Efficient Context Parallel Training (NeurIPS 2025 · similarity 0.87)
- Harnessing Manycore Processors with Distributed Memory for Accelerated Training of Sparse and Recurrent Models (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.