Research map / Architectures and efficiency / Attention and transformers

Attention-based PCA

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Sparse and structured attention · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that rank-one softmax and linear attention trained on Gaussian tokens with a population reconstruction risk learn the top principal eigenvector of the covariance, analyzed for infinite and finite prompts and spiked Wishart in-context settings.

Problem

Understanding what unsupervised objectives attention layers optimize, and whether attention can recover principal components of Gaussian data in finite and infinite prompt regimes.

Approach

Reduces a rank-one softmax attention layer on Gaussian prompts to a linear operator in the infinite-prompt limit, characterizes its loss landscape, shows gradient flow converges to the leading eigenvector, transfers this to finite prompts via uniform convergence of risks and derivatives, and links the dynamics to Oja's flow.

Key results

• Infinite-prompt softmax risk has global minimizers at ±u_1/sqrt(λσ_1); gradient flow converges to them for almost every initialization.
• Finite-prompt risk gradient flow converges uniformly on compact time intervals to the infinite-prompt trajectory as L grows; W2^2 between finite- and infinite-prompt output laws is O(L^{-ε}(1+ln L)^{1-ε}).
• In the spiked Wishart ICL setting, the only local minima are ±α* v, the spike direction, with convergence rate scaling with ξ², θ and n.

Limitations

• Simplified rank-one attention with identity value matrix and population (not empirical) risk training.
• Finite-prompt rates from theory are conservative relative to observed practice.
• Results stated for Gaussian tokens with fixed covariance; the ICL extension relies on Wishart-distributed covariances.

Experimental setup

Datasets: synthetic Gaussian tokens spiked Wishart covariances

Metrics: alignment with principal eigenvector population risk convergence rate 2-Wasserstein distance

Baselines: linear attention

Builds on: Oja's rule measure-based attention formalism Stable Manifold Theorem

Closest papers by meaning

• Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysis (NeurIPS 2024 · similarity 0.89)
• Rethinking PCA Through Duality (NeurIPS 2025 · similarity 0.88)
• Mind the Gap: a Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers (ICML 2025 · similarity 0.86)
• Bayes optimal learning of attention-indexed models (NeurIPS 2025 · similarity 0.85)
• In-Context Linear Regression Demystified: Training Dynamics and Mechanistic Interpretability of Multi-Head Softmax Attention (ICML 2025 · similarity 0.85)
• Degrees of Freedom for Linear Attention: Distilling Softmax Attention with Optimal Feature Efficiency (NeurIPS 2025 · similarity 0.85)