Research map / Optimization / Optimizers

Factor Augmented High-Dimensional SGD

NeurIPS 2026 · Optimization › Optimizers · approach: SGD momentum and step-size schedules · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Factor-Augmented SGD (FSGD), which learns latent factors online via Oja's streaming PCA while running SGD on the factor-projected inputs, and gives l^s-norm moment convergence bounds that account for factor estimation error and idiosyncratic noise.

Problem

SGD in high-dimensional feature spaces is computationally costly and theoretically under-understood; existing factor-augmented methods use a two-stage offline PCA that needs full data access and ignores how factor estimation error affects SGD convergence.

Approach

Couple a streaming Oja PCA update with mini-batch SGD on the estimated factors, so factors are refined during training rather than fixed beforehand; analyze the resulting drift of the population minimizer (minimizer shift) and the factor estimation error in l^s-norm moment bounds.

Key results

• On synthetic neural network experiments at d=400, Factor-NN reaches test L2 loss 0.086 vs 0.087 for Oracle-NN and 0.387 for Vanilla-NN (with standard deviations reported).
• On NCEP real data, FSGD achieves test R^2 of 0.641 on P06_TropSPac4 vs 0.631 for PPCA and 0.549 for Vanilla SGD; FSGD is best in three of four response locations.
• FSGD uses about 0.40 MB storage at k=5 on NCEP, versus 9.62 MB for PPCA with window W=120.

Surprising findings

• FSGD-based methods sometimes outperform the Oracle-NN (true-factor) estimator when the ambient dimension is large, attributed to implicit regularization from the projection.
• Continued Oja updates in Stage II make FSGD insensitive to warm-up length, even with T0=1, whereas freezing the projection right after warm-up degrades MSE severely (0.53 to 1.49).

Limitations

• Theory is developed for strongly convex population objectives, while neural network experiments are non-convex and only empirically supported.
• Theoretical bounds include a minimizer-drift term that does not benefit from larger learning rates, requiring a freezing strategy to mitigate.
• Real-data evaluation is limited to a linear predictor on one dataset and does not beat PPCA on every location (PPCA better on P01_SubNAtl10).

Experimental setup

Datasets: NCEP/NCAR Reanalysis 1

Benchmarks: NCEP/NCAR Reanalysis 1 (Z500 anomalies)

Metrics: test L2 loss R^2 per-step runtime (ms) memory (MB) parameter error |theta_T - theta*_{T-1}|_s

Baselines: Oracle-NN Vanilla-NN PCA-A-NN NN-Joint Random Proj PPCA Vanilla SGD Persistence

Builds on: Oja's algorithm factor models stochastic gradient descent

Closest papers by meaning

• Statistical Guarantees for High-Dimensional Stochastic Gradient Descent (NeurIPS 2025 · similarity 0.87)
• Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensions (NeurIPS 2022 · similarity 0.87)
• A Variational Analysis of Stochastic Gradient Algorithms (ICML 2016 · similarity 0.86)
• The Benefits of Implicit Regularization from SGD in Least Squares Problems (NeurIPS 2021 · similarity 0.86)
• Risk Bounds of Accelerated SGD for Overparameterized Linear Regression (ICLR 2024 · similarity 0.84)
• Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetries (ICLR 2025 · similarity 0.84)