Research map / Learning theory and algorithms / Kernels and regression theory
Optimality and Adaptivity of Deep Neural Features for Instrumental Variable Regression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives minimax-optimal convergence rates for deep feature instrumental variable (DFIV) regression with sigmoid DNN features in Besov spaces, and shows separation from fixed-feature linear IV estimators when p<2.
Problem
Convergence rates and minimax optimality of data-adaptive deep-neural-feature two-stage IV (DFIV) regression for nonparametric IV, and whether it beats fixed-feature (kernel/sieve) IV estimators.
Approach
Analyzes DFIV with learnable sigmoid DNN features in both stages, using a Besov smoothness assumption on T and a link/reverse-link condition framework; proves DNN approximation for Besov functions with sigmoid nets and derives projected and full rates, plus lower bounds for linear IV estimators showing separation.
Key results
- DFIV attains the nearly minimax optimal rate n^{-(2s+2γ0)/(2s+2γ0+d_x)} (log n) in the projected metric when m=Ω(n), γ0=γ1 and T has maximal smoothness.
- Full-metric DFIV upper bound matches the NPIV lower bound n^{-2s/(2s+2γ1+d_x)} up to logs for p≥2, requiring only m=Ω(n) Stage 1 samples versus m/n→∞ for kernel IV.
- For p<2, any linear IV estimator is lower bounded by n^{-2(s-Δ)/(2(s-Δ)+2γ1+d_x)} with Δ=d_x(1/p-1/2)_+, which DFIV can beat under an extended link condition.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use adaptive deep neural features instead of fixed features | DFIV vs fixed-feature IV · NPIV Besov with low spatial homogeneity (p<2) | full MSE rate: DFIV: n^{-\frac{2s}{2s+2\gamma_1+d_x}}; fixed-feature IV: n^{-\frac{2(s-\Delta)}{2(s-\Delta)+2\gamma_1+d_x}} | Helped |
| use Stage 1 sample size m=\Omega(n) | DFIV · NPIV, minimax-optimal projected and full rates | sample requirement: m= \Omega(n); kernel IV estimators require m/n\to\infty | Helped |
| use DFIV under the stated conditions (\gamma_0=\gamma_1 and maximal smoothness of T) | DFIV · NPIV Besov, projected error | projected MSE rate: n^{-\frac{2s+2\gamma_0}{2s+2\gamma_0+d_x}}\log n | Helped |
| use DFIV under the stated conditions (p\ge 2, \gamma_0=\gamma_1, and maximal smoothness of T) | DFIV · NPIV Besov, full error | full MSE rate: n^{-\frac{2s}{2s+2\gamma_1+d_x}}(\log n)^\frac{s}{s+\gamma_1} | Helped |
Surprising findings
- The Stage 2 rate depends only on the Stage 1 smoothness s', while the Stage 1 rate depends on smoothness of both stages.
- DFIV needs only Stage 1 sample size m=Ω(n), unlike kernel IV which requires m/n→∞ for minimax rates.
Limitations
- Smoothness assumption on T (Assumption on T mapping Besov to Besov) and domain restriction or Besov-norm regularization are needed for the rate and can be restrictive.
- Analysis is theoretical with no experiments; the lower bound for p<2 requires extended reverse link condition.
- Results for ReLU DNNs carry extra log factors and the theory is stated for sigmoid activations.
Experimental setup
Metrics: minimax convergence rate (projected and full L2 MSE)
Baselines: kernel IV sieve IV 2SLS Nadaraya-Watson IV
Builds on: DFIV 2SLS NPIV link condition B-spline Besov approximation
Closest papers by meaning
- Nonparametric Instrumental Variable Regression through Stochastic Approximate Gradients (NeurIPS 2024 · similarity 0.85)
- Demystifying Spectral Feature Learning for Instrumental Variable Regression (NeurIPS 2025 · similarity 0.85)
- Learning Decision Policies with Instrumental Variables through Double Machine Learning (ICML 2024 · similarity 0.84)
- Stochastic Optimization Algorithms for Instrumental Variable Regression with Streaming Data (NeurIPS 2024 · similarity 0.84)
- Out of the Ordinary: Spectrally Adapting Regression for Covariate Shift (ICML 2024 · similarity 0.84)
- Geometry-Aware Instrumental Variable Regression (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.