Research map / Optimization / Nonconvex and smooth optimization
Stochastic Optimization Algorithms for Instrumental Variable Regression with Streaming Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes streaming SGD-style algorithms that solve least-squares instrumental variable regression directly as a conditional stochastic optimization problem, without mini-batches or matrix inversion, with O(log T/T) rate under a two-sample oracle and O(1/T^{1-ι}) under a one-sample oracle.
Problem
Instrumental variable regression with streaming data: existing methods either use forbidden two-stage regression or reformulate the conditional stochastic optimization objective as a minimax problem with approximation error, and nested-sampling methods need mini-batches.
Approach
Directly optimize the conditional stochastic objective E_Z E_{Y|Z}[(Y - E_{X|Z}[g(X)])^2]; with a two-sample oracle (two conditionally independent X, X' given Z) the gradient product of two conditional expectations has an unbiased single-sample estimator, avoiding nuisance modeling. The one-sample variant jointly runs an online estimate of the first-stage coefficient and uses its prediction in pla
Key results
- TOSG-IVaR (two-sample) achieves O(log T / T) last-iterate expected error rate for linear models under stated moment assumptions.
- OTSG-IVaR (one-sample) achieves O(1/T^{1-ι}) rate for any ι>0 under linear X-Z model.
- In simulations, OTSG-IVaR shows consistently stable performance across settings where O2SLS exhibits much larger variance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the stabilized OTSG-IVaR θ update, replacing X_t^T θ_t with Z_t^T γ_t θ_t | OTSG-IVaR · d_x=d_z=1, θ*=1, γ*=-1, ρ=4, σ_ε=1; γ_0=10, θ_0=0 | parameter error: ≈ 10^4 -> ≈ 10^{-5} | Helped |
| use the CSO — Eq. (11) updates instead of the OTSG-IVaR updates (ablation) | OTSG-IVaR · one-sample IVaR simulation, d_x=d_z=1; γ_0=10, θ_0=0 | finite-time parameter error: ≈ 10^{-5} vs ≈ 10^4 | Hurt |
| use O2SLS rather than the proposed algorithms | one-sample IVaR simulation · ρ=1,4; σ_ε=0.5,1; (d_x,d_z)=(1,1),(8,16) | variance of parameter error across trials | Hurt |
Surprising findings
- The two-stage O2SLS-style update with gamma_t Z_t Z_t^T gamma_t theta_t interaction can diverge initially before converging, motivating the modified update using Z_t^T gamma_t.
Limitations
- Theoretical guarantees are for linear models; the nonlinear neural-network case only has local results under stronger assumptions (Proposition with O(1/sqrt T) gradient norm bound).
- Requires the two-sample oracle for the O(log T/T) rate, which may be unavailable; assumes i.i.d. streaming data.
- Analysis assumes bounded compact iterates for gamma_t (Assumption on compact set) and moment conditions rather than standard bounded variance.
Experimental setup
Datasets: synthetic linear and nonlinear IV data generated from Gaussian models
Benchmarks: synthetic IV simulations
Metrics: expected squared parameter error ||theta_t - theta*||^2 test MSE
Baselines: O2SLS (Ai et al. online 2SLS) DVB23 minimax/CSO algorithm (Dai et al.) Eq. (11) CSO variant
Builds on: two-stage least squares O2SLS conditional stochastic optimization
Closest papers by meaning
- Nonparametric Instrumental Variable Regression through Stochastic Approximate Gradients (NeurIPS 2024 · similarity 0.89)
- Stochastic Online Instrumental Variable Regression: Regrets for Endogeneity and Bandit Feedback (AAAI 2025 · similarity 0.88)
- Conditional Instrumental Variable Regression with Representation Learning for Causal Inference (ICLR 2024 · similarity 0.86)
- Demystifying Spectral Feature Learning for Instrumental Variable Regression (NeurIPS 2025 · similarity 0.86)
- Geometry-Aware Instrumental Variable Regression (ICML 2024 · similarity 0.85)
- Optimality and Adaptivity of Deep Neural Features for Instrumental Variable Regression (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.