Research map / Optimization / Optimizers
Randomized Automatic Differentiation
Deniz Oktay, Nick McGreivy, Joshua Aduol, Alex Beatson, Ryan P Adams
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use activation sampling with a large mini-batch instead of matching memory by reducing batch size | RAD feedforward networks · CIFAR-10 and MNIST | training loss convergence | Improved |
| Use activation sampling with a large mini-batch instead of reducing batch size | RAD · Sequential-MNIST | training loss convergence | No clear effect |
| Sample different activations for each mini-batch element rather than reusing the same sample (ablation) | RAD fully connected network · MNIST | training loss convergence | Improved |
| Sample different activations for each mini-batch element rather than reusing the same sample (ablation) | RAD convolutional network · CIFAR-10 | training loss convergence | No clear effect |
| Use a random projection instead of activation sampling (ablation) | RAD · MNIST and CIFAR-10 | training performance | No clear effect |
| Use an activation fraction of 0.1 instead of the full graph (ablation) | RAD · ConvNet training | peak memory (MBytes): 23.08 -> 3.28 | Improved |
| Use an activation fraction of 0.1 instead of the full graph (ablation) | RAD · MNIST fully connected training | peak memory (MBytes): 2.69 -> 1.80 | Improved |
| Use an activation fraction of 0.1 instead of the full graph (ablation) | RAD · Sequential-MNIST RNN training | peak memory (MBytes): 47.93 -> 7.01 | Improved |
| Randomize intermediate Jacobians without tailoring sampling to the graph structure | RAD neural networks · Deep neural networks | optimization convergence | Hurt |
| Apply RAD to selected gradient factors in the PDE estimator | Reaction-diffusion PDE optimization · Linear reaction-diffusion PDE | memory reduction: 99% | Improved |
Closest papers by meaning
- Storchastic: A Framework for General Stochastic Automatic Differentiation (NeurIPS 2021 · similarity 0.88)
- Automatic Differentiation in Machine Learning: a Survey (JMLR 2018 · similarity 0.87)
- Stochastic Taylor Derivative Estimator: Efficient amortization for arbitrary differential operators (NeurIPS 2024 · similarity 0.86)
- Collapsing Taylor Mode Automatic Differentiation (NeurIPS 2025 · similarity 0.85)
- Statistical Guarantees for High-Dimensional Stochastic Gradient Descent (NeurIPS 2025 · similarity 0.85)
- Non-asymptotic Analysis of Biased Adaptive Stochastic Approximation (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
