Research map / Architectures and efficiency / RNNs and normalization
PCA of high dimensional random walks with comparison to neural network training
Joseph Antognini, Jascha Sohl-Dickstein
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PCA to examine a high-dimensional random walk trajectory | High-dimensional random walk · 10,000 dimensions, 1000 steps | PCA explained variance distribution | No clear effect |
| Use non-isotropic Gaussian noise instead of isotropic noise | High-dimensional random walk · 1000 dimensions, 1000 steps | PCA explained variance distribution | No clear effect |
| Add momentum to the random walk | High-dimensional random walk · 10,000 dimensions, 1000 steps | PCA variance distribution | Mixed |
| Increase momentum to γ = 0.999 | High-dimensional random walk · 10,000 dimensions, 1000 steps | PCA variance distribution | Hurt |
| Increase the Ornstein-Uhlenbeck potential strength | High-dimensional Ornstein-Uhlenbeck process · 30,000 dimensions, 3000 steps | PCA eigenvalue distribution | Mixed |
| Use exponentially decaying step sizes | High-dimensional random walk · Synthetic Gaussian linear regression trajectory | PCA-projected trajectory similarity | Improved |
| Train for longer than the crossing time before using iterate averages | Ornstein-Uhlenbeck process · Converged high-dimensional process | Estimate of the minimum | Improved |
| Compare neural-network parameter trajectories with a high-dimensional random walk | ResNet-50-v2 · Imagenet training trajectory | PCA-projected trajectory similarity | Mixed |
| Compare a linear model's PCA variance distribution across training phases | Linear regression · CIFAR-10 training trajectory | PCA variance distribution | Mixed |
Closest papers by meaning
- Towards Understanding Transformers in Learning Random Walks (NeurIPS 2025 · similarity 0.83)
- What kills $v$-prediction? A Patch-wise PCA Perspective on Pixel-Space Flow Matching (NeurIPS 2026 · similarity 0.83)
- Precise expressions for random projections: Low-rank approximation and randomized Newton (NeurIPS 2020 · similarity 0.83)
- Random Matrix Theory of Early-Stopped Gradient Flow: A Transient BBP Scenario (NeurIPS 2026 · similarity 0.82)
- Random features for Grassmannian kernel approximation with bounded rank-one projections (NeurIPS 2026 · similarity 0.82)
- The High Line: Exact Risk and Learning Rate Curves of Stochastic Adaptive Learning Rate Algorithms (NeurIPS 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
