Research map / Optimization / Optimizers
Improving Neural Network Training in Low Dimensional Random Bases
Frithjof Gressmann, Zach Eaton-Rosen, Carlo Luschi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Re-draw the random subspace at every step (RBD) instead of a fixed random projection (FPD) (ablation) | random subspace training d=250 · CNN-CIFAR10 | validation accuracy (%): 37.2 -> 54.64 | Improved |
| Re-draw the random subspace at every step (RBD) instead of a fixed projection (ablation) | random subspace training d=250 · FC-CIFAR10 | validation accuracy (%): 21.4 -> 43.77 | Improved |
| Re-draw the random subspace at every step (RBD) instead of a fixed projection (ablation) | random subspace training d=250 · FC-MNIST | validation accuracy (%): 80 -> 93.893 | Improved |
| Use RBD instead of evolution-strategy black-box optimization (NES) at same dimensionality (ablation) | random subspace training d=250 · FC-MNIST | validation accuracy (%): 22.5 -> 93.893 | Improved |
| Increase the number of random directions (reduction factor from 75x to 10x) (ablation) | RBD · reduced CIFAR-10 network | validation accuracy (%): 62.44 -> 70.26 | Improved |
| Train longer (2000 epochs) with only d=2 directions (ablation) | RBD · CNN-CIFAR10 | validation accuracy (%): 68.06 -> 48.20 | Hurt |
| Use more directions (10 and 50) instead of 2 at convergence (ablation) | RBD · CNN-CIFAR10 | validation accuracy (%): 48.20 -> higher than 2-direction limit | Improved |
| Replace Gaussian bases with Uniform bases (ablation) | RBD · CNN-CIFAR10 | validation accuracy (%): 52.3 -> 41 | Hurt |
| Replace Gaussian bases with Bernoulli-0.5 bases (ablation) | RBD · CNN-CIFAR10 | validation accuracy (%): 52.3 -> 28.5 | Hurt |
| Replace Gaussian bases with Bernoulli-0.5 bases (ablation) | RBD · FC-MNIST | validation accuracy (%): 94.95 -> 77.5 | Hurt |
| Switch between RBD and SGD during training without tuning learning rates (ablation) | RBD/SGD hybrid · CIFAR10-CNN | validation accuracy | No clear effect |
| Start with SGD then switch to RBD (ablation) | RBD after SGD · CIFAR10-CNN | validation accuracy | Hurt |
Closest papers by meaning
- How many degrees of freedom do we need to train deep networks: a loss landscape perspective (ICLR 2022 · similarity 0.89)
- Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Training (JMLR 2022 · similarity 0.89)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.87)
- Does SGD really happen in tiny subspaces? (ICLR 2025 · similarity 0.87)
- Random Sparse Lifts: Construction, Analysis and Convergence of finite sparse networks (ICLR 2024 · similarity 0.87)
- The Power of Random Features and the Limits of Distribution-Free Gradient Descent (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
