Research map / Architectures and efficiency / Attention and transformers

Ordinary Least Squares as an Attention Mechanism

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Sparse and structured attention · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that OLS predictions can be rewritten as a restricted (identity-activation) attention module with embedding (X'X)^{-1}, extends it to PCR, ridge, softmax and stacking, and proposes a Regression Block that replaces the attention sublayer of FT-Transformer with polynomial-feature regression.

Problem

Understanding the relationship between attention and classical least squares regression, and whether the attention sublayer of a tabular transformer can be replaced by explicit regression on expanded features.

Approach

Reframes OLS out-of-sample predictions as inner-product similarity weighting of training outcomes in an orthonormalized predictor space, which is exactly linear attention with W_Q W_K' = (X'X)^{-1} and identity activation; proposes Attention Regression (nonlinear, softmax-free) and a Regression Block that replaces softmax attention with degree-2 polynomial cross-features plus PCA and linear regres

Key results

• Monte Carlo over 96 conditions: Attention Regression averages R^2=0.48, vs MLP 0.52, Random Forest 0.45, Gradient Boosting 0.42, OLS 0.28.
• On eight UCI/OpenML datasets, the Regression Block is ahead of FT-Transformer on all eight datasets (e.g., Kin8nm 0.920 vs 0.914) at roughly 30% of the parameters (about 30K vs 100K).
• Six of the eight Regression Block margins over FT-Transformer are reported as within seed noise.

Surprising findings

• The Regression Block, with no softmax attention, matches or slightly beats FT-Transformer on all eight tabular datasets while using about 30% of the parameters.
• Softmax attention in the same skeleton performs far behind the explicit polynomial regression, suggesting the explicit regression carries the result.

Limitations

• The exact identity covers one restricted attention module; in full transformers the reading is local to each module.
• Regression Block cross-features grow quadratically with number of inputs, becoming binding for high-dimensional data.
• Attention Regression with five heads is only a proof of concept and trails MLP and Random Forest on average.

Experimental setup

Datasets: California Yacht Energy Concrete Airfoil Abalone Kin8nm Protein

Benchmarks: UCI/OpenML tabular regression (8 datasets) Monte Carlo simulations (6 DGPs)

Metrics: out-of-sample R^2 accuracy

Baselines: OLS Random Forest Gradient Boosting MLP FT-Transformer

Builds on: Transformer self-attention FT-Transformer principal component regression ridge regression representer theorem

Closest papers by meaning

• Attention layers provably solve single-location regression (ICLR 2025 · similarity 0.87)
• In-Context Linear Regression Demystified: Training Dynamics and Mechanistic Interpretability of Multi-Head Softmax Attention (ICML 2025 · similarity 0.86)
• Sparse Attention as Compact Kernel Regression (NeurIPS 2026 · similarity 0.86)
• Revisiting Attention for Multivariate Time Series Forecasting (AAAI 2025 · similarity 0.86)
• Degrees of Freedom for Linear Attention: Distilling Softmax Attention with Optimal Feature Efficiency (NeurIPS 2025 · similarity 0.84)
• Attention-based PCA (NeurIPS 2026 · similarity 0.84)