Research map / Optimization / Optimizers
Stochastic Gradients under Nuisances
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives non-asymptotic convergence guarantees for SGD with a plug-in unknown nuisance, showing Neyman orthogonality yields fourth-power nuisance bias, and proposes OSGD which uses an estimated orthogonalizing operator to interpolate between regimes.
Problem
Stochastic gradient descent on risk minimization problems that depend on an unknown (estimated) nuisance parameter lacks convergence guarantees, especially for orthogonal statistical learning and double machine learning settings.
Approach
Analyzes SGD run with a plug-in nuisance estimate: under Neyman orthogonality the nuisance bias scales with the fourth power of the nuisance error instead of the square; for non-orthogonal losses, an orthogonalized gradient oracle (OSGD) subtracts an estimated orthogonalizing operator times the nuisance gradient, interpolating between the two rates depending on operator accuracy.
Key results
- SGD under Neyman orthogonality converges linearly to a neighborhood of theta_star with bias radius proportional to ||g_hat - g_0||^4 instead of ||g_hat - g_0||^2 in the non-orthogonal case.
- OSGD's bias term includes ||g_hat - g_0||^2 times ||Gamma_hat - Gamma_0||_Fro^2, so it matches the nuisance-insensitive rate when the orthogonalizer error is O(||g_hat - g_0||).
- For a nonparametric nuisance with polynomial eigen-decay, the nuisance bias for non-orthogonal losses is O(m^{-(2alpha-1)/(2alpha)}), and OSGD improves this to O_p(m^{-(2alpha-1)/alpha}).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a Neyman-orthogonal loss instead of a non-orthogonal loss | SGD with an estimated nuisance · Unknown-nuisance stochastic optimization | nuisance-dependent term in expected squared distance to optimum: \frac{2\alpha_1^2}{\mu^2} \Vert\hat g - g_0\Vert_\G^2 -> \frac{\beta_1^2}{2\mu^2} \Vert\hat g - g_0\Vert_\G^4 | Helped |
| use an approximately orthogonalized gradient oracle with an estimated orthogonalizing operator | OSGD · Unknown-nuisance stochastic optimization | nuisance-dependent term in expected squared distance to optimum: \beta_2^2\gnorm{\hat g - g_0}^4 + 4\alpha_2^2\gnorm{\hat g - g_0}^2\cdot\norm{\hat\Gamma - \Gamma_0}_\mathrm{Fro}^2 | Helped |
| estimate the orthogonalizing operator with error of order the nuisance-estimation error | OSGD · Unknown-nuisance stochastic optimization | nuisance-dependent convergence rate: \Vert\hat g - g_0\Vert_\G^4 + \Vert\hat g - g_0\Vert_\G^2 \cdot \norm{\hat\Gamma - \Gamma_0}_{\mathrm{Fro}}^2 | Helped |
Limitations
- Constants depend on the initial nuisance radius r and bounded-moment assumptions; learning rate must be small.
- Results are for single-sample SGD with unbiased oracles; extensions to other optimizers only discussed in appendix.
- Estimation of the orthogonalizing operator Gamma_0 is needed and its quality governs OSGD's gain.
Experimental setup
Metrics: mean squared distance to optimum E||theta_n - theta_star||^2 convergence rate in nuisance error
Baselines: full-batch empirical risk minimization with plug-in nuisance biased SGD analyses
Builds on: double machine learning orthogonal statistical learning Neyman orthogonality
Closest papers by meaning
- High-probability Convergence of Gradient Methods under Markovian Stochasticity (NeurIPS 2026 · similarity 0.89)
- Universality of AdaGrad Stepsizes for Stochastic Optimization: Inexact Oracle, Acceleration and Variance Reduction (NeurIPS 2024 · similarity 0.89)
- Simple Stochastic Gradient Methods for Non-Smooth Non-Convex Regularized Optimization (ICML 2019 · similarity 0.88)
- Decomposable Non-Smooth Convex Optimization with Nearly-Linear Gradient Oracle Complexity (NeurIPS 2022 · similarity 0.88)
- On the Convergence of mSGD and AdaGrad for Stochastic Optimization (ICLR 2022 · similarity 0.87)
- Universal Gradient Methods for Stochastic Convex Optimization (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.