Research map / Architectures and efficiency / RNNs and normalization
DeepPCR: Parallelizing Sequential Operations in Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use smoother activation (sigmoid/tanh) instead of ReLU (ablation) | DeepPCR forward pass in MLPs · MLP forward pass break-even depth | break-even depth L: 2^7 -> 2^6 | Helped |
| apply DeepPCR to backward pass instead of forward pass | DeepPCR on MLP · MLP break-even depth | break-even depth L: 2^7 -> 2^5 | Helped |
| stop Newton solver at 3 iterations (ablation) | DeepPCR ResNet training · MNIST ResNet | speedup: 7 -> 14 | Helped |
| use 80GB A100 instead of 40GB V100 (ablation) | DeepPCR on MLP · MLP forward pass, large w^2L | depth before linear regime: delayed by a factor of 2 | Helped |
| increase MLP width past threshold | DeepPCR on MLP · MLP forward pass | speedup | Hurt |
| increase latent size w past 2^6 at L=2^10 | DeepPCR latent diffusion on MNIST · MNIST latent diffusion | speedup | Hurt |
| train ResNet with DeepPCR instead of sequential passes | MNIST ResNet · MNIST test accuracy | accuracy (%): 94 -> 94 | No clear effect |
| switch from ReLU to smoother activations (ablation) | Newton solver in DeepPCR · MLP forward pass | Newton iterations to converge | Helped |
| apply DeepPCR to CIFAR-10 and CelebA latent diffusion instead of MNIST | DeepPCR latent diffusion · latent diffusion speedup | speedup: 11.2 -> 8 (CIFAR-10), 9 (CelebA) | Mixed |
Closest papers by meaning
- Parallelizing non-linear sequential models over the sequence length (ICLR 2024 · similarity 0.87)
- Mesh-TensorFlow: Deep Learning for Supercomputers (NeurIPS 2018 · similarity 0.87)
- ASPEN: Breaking Operator Barriers for Efficient Parallelization of Deep Neural Networks (NeurIPS 2023 · similarity 0.86)
- $\mu$PC: Scaling Predictive Coding to 100+ Layer Networks (NeurIPS 2025 · similarity 0.86)
- Pfeife: Automatic Pipeline Parallelism for PyTorch (ICML 2025 · similarity 0.85)
- Encoder-Decoder Diffusion Language Models for Efficient Training and Inference (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.