Research map / Optimization / Implicit bias and generalization
Transformers Handle Endogeneity in In-Context Linear Regression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that transformers can implement a gradient-based bi-level optimization converging exponentially to 2SLS for in-context IV regression, and proves an excess-loss bound for an ICL-pretrained looped transformer that empirically matches or beats 2SLS under endogeneity.
Problem
In-context linear regression under endogeneity (noise correlated with regressors), where OLS is biased and inconsistent; whether transformers can use instrumental variables to give reliable predictions and coefficient estimates.
Approach
Constructs a looped attention-only transformer that emulates a bi-level GD procedure converging exponentially to the 2SLS estimator, paired with an ICL pretraining scheme where endogeneity occurs in training samples but not the query, plus a coefficient-extraction method via output differentiation.
Key results
- Under standard endogeneity the trained transformer performs close to 2SLS and better than OLS on ICPE and coefficient MSE (averaged over 500 simulations).
- With weak instruments (IV strength < 0.5) the trained transformer outperforms 2SLS.
- In non-standard settings (quadratic IV effect, underdetermined IV with q=3, p=5) the transformer consistently beats both 2SLS and OLS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the ICL-pretrained transformer instead of 2SLS | ICL-pretrained transformer · Standard endogeneity tasks | ICPE | No clear effect |
| Use the ICL-pretrained transformer instead of 2SLS | ICL-pretrained transformer · Standard endogeneity tasks | coefficient MSE | No clear effect |
| Use 2SLS instead of OLS | 2SLS · Standard endogeneity tasks | ICPE and coefficient MSE | Helped |
| Use the ICL-pretrained transformer instead of 2SLS | ICL-pretrained transformer · Weak instruments (IV strength < 0.5) | ICPE and coefficient MSE | Helped |
| Use the ICL-pretrained transformer instead of 2SLS | ICL-pretrained transformer · Strong instruments | ICPE and coefficient MSE | No clear effect |
| Use the ICL-pretrained transformer instead of 2SLS and OLS | ICL-pretrained transformer · Quadratic-effect non-linear IV | ICPE and coefficient MSE | Helped |
| Use the ICL-pretrained transformer instead of 2SLS and OLS | ICL-pretrained transformer · Underdetermined IV (q=3, p=5) | ICPE and coefficient MSE | Helped |
Surprising findings
- The trained transformer outperforms 2SLS on weak instruments, suggesting ICL training yields a mechanism beyond mimicking 2SLS.
- The paper's own bound and experiments suggest a gap remains between 2SLS and the optimal predictor that the transformer bridges.
Limitations
- Experiments are purely simulated synthetic linear/IV settings with clipping disabled.
- Theoretical bounds assume bounded variables and linear instruments; relaxations to sub-Gaussian noise are only anticipated.
Experimental setup
Datasets: synthetic in-context linear regression prompts (Algorithm 1)
Benchmarks: synthetic endogenous linear regression tasks
Metrics: in-context prediction error (ICPE) coefficient MSE
Baselines: 2SLS OLS
Builds on: looped transformer transformers implement gradient descent for in-context linear regression two-stage least squares
Closest papers by meaning
- Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regression (NeurIPS 2024 · similarity 0.89)
- Linear Transformers are Versatile In-Context Learners (NeurIPS 2024 · similarity 0.86)
- One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attention (ICLR 2024 · similarity 0.86)
- Does learning the right latent variables necessarily improve in-context learning? (ICML 2025 · similarity 0.86)
- On Mesa-Optimization in Autoregressively Trained Transformers: Emergence and Capability (NeurIPS 2024 · similarity 0.86)
- Transformers can optimally learn regression mixture models (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.