AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

PCA of high dimensional random walks with comparison to neural network training

Joseph Antognini, Jascha Sohl-Dickstein

NeurIPS 2018 · Architectures and efficiency › RNNs and normalization · approach: Symmetry and equivariance in loss landscapes · 11 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use PCA to examine a high-dimensional random walk trajectoryHigh-dimensional random walk · 10,000 dimensions, 1000 stepsPCA explained variance distributionNo clear effect
Use non-isotropic Gaussian noise instead of isotropic noiseHigh-dimensional random walk · 1000 dimensions, 1000 stepsPCA explained variance distributionNo clear effect
Add momentum to the random walkHigh-dimensional random walk · 10,000 dimensions, 1000 stepsPCA variance distributionMixed
Increase momentum to γ = 0.999High-dimensional random walk · 10,000 dimensions, 1000 stepsPCA variance distributionHurt
Increase the Ornstein-Uhlenbeck potential strengthHigh-dimensional Ornstein-Uhlenbeck process · 30,000 dimensions, 3000 stepsPCA eigenvalue distributionMixed
Use exponentially decaying step sizesHigh-dimensional random walk · Synthetic Gaussian linear regression trajectoryPCA-projected trajectory similarityImproved
Train for longer than the crossing time before using iterate averagesOrnstein-Uhlenbeck process · Converged high-dimensional processEstimate of the minimumImproved
Compare neural-network parameter trajectories with a high-dimensional random walkResNet-50-v2 · Imagenet training trajectoryPCA-projected trajectory similarityMixed
Compare a linear model's PCA variance distribution across training phasesLinear regression · CIFAR-10 training trajectoryPCA variance distributionMixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy