Research map / Architectures and efficiency / Attention and transformers
Attention boosted Individualized Regression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Attention boosted Individualized Regression (AIR), a matrix-input regression with sample-specific coefficients derived from an internal patch-relation map that links it to linear self-attention, and gives alternating-minimization convergence guarantees.
Problem
Individualized regression for matrix-valued inputs usually needs prior sample-similarity information; the paper aims to get sample-specific coefficients from the internal relations among patches within each sample, motivated by brain MRI analysis.
Approach
Decompose each sample's coefficient into a homogeneous part plus a heterogeneous part obtained by re-weighting shared base coefficients with a sample-specific patch-relation matrix g(X_i W X_i^T), which is shown to be equivalent to linear self-attention; fitted by alternating minimization with spectral initialization.
Key results
- In simulations (Case 1, DI=0.5) AIR achieves RMSE 4.422 vs 4.886 for DKN and 18.429 for ViT; advantage grows with DI (DI=2.0 Case 2: 13.002 vs 28.686 for DKN).
- On ADNI 5-fold CV, AIR achieves test RMSE 3.145 (0.019), best among LRMR 3.715, TRLasso 3.292, DKN 3.261, ViT 3.282.
- Theory: alternating minimization converges geometrically to the true parameters under a 2-RIP condition with spectral initialization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AIR instead of DKN | Simulation Case 1, DI=0.5 · Simulation Case 1, DI=0.5 | test RMSE: 4.886 (0.018) -> 4.422 (0.130) | Helped |
| Use AIR instead of DKN | Simulation Case 1, DI=1.0 · Simulation Case 1, DI=1.0 | test RMSE: 7.028 (0.032) -> 8.102 (0.325) | Hurt |
| Use AIR instead of DKN | Simulation Case 1, DI=2.0 · Simulation Case 1, DI=2.0 | test RMSE: 11.741 (0.043) -> 10.599 (0.816) | Helped |
| Use AIR instead of LRMR | Simulation Case 2, DI=0.5 · Simulation Case 2, DI=0.5 | test RMSE: 6.766 (0.018) -> 3.590 (0.046) | Helped |
| Use AIR instead of LRMR | Simulation Case 2, DI=1.0 · Simulation Case 2, DI=1.0 | test RMSE: 13.408 (0.037) -> 6.632 (0.022) | Helped |
| Use AIR instead of LRMR | Simulation Case 2, DI=2.0 · Simulation Case 2, DI=2.0 | test RMSE: 26.864 (0.074) -> 13.002 (0.044) | Helped |
| Use AIR instead of TRLasso | ADNI · ADNI | test RMSE: 3.292 (0.023) -> 3.145 (0.019) | Helped |
| Add the homogeneous part to the heterogeneous-only model (ablation) | AIR · Simulation Case 1, DI varying from 1/4 to 4 | test RMSE | Helped |
| Add the heterogeneous part to the homogeneous-only model (ablation) | AIR · Simulation Case 1, DI varying from 1/4 to 4 | test RMSE | Helped |
| Add the homogeneous part to the heterogeneous-only model (ablation) | AIR · Simulation Case 2, DI varying from 1/4 to 4 | test RMSE | Helped |
| Add the heterogeneous part to the homogeneous-only model (ablation) | AIR · Simulation Case 2, DI varying from 1/4 to 4 | test RMSE | Helped |
Surprising findings
- ViT performs worst in simulations (e.g., 18.4 to 44.0 RMSE) due to limited sample size, far behind even linear baselines.
Limitations
- Performance reduces to an ordinary linear model when heterogeneous effects are minimal.
- Linear form only; the framework is a simplified ViT and may lose approximation power on complex tasks.
- Theory relies on RIP and low-rank assumptions on W and D.
Experimental setup
Datasets: ADNI (1059 subjects, brain MRI with MMSE scores) simulated 28x28 matrix data
Benchmarks: simulated individualized regression (DI 0.5, 1.0, 2.0) ADNI MMSE prediction
Metrics: RMSE test RMSE standard error
Baselines: LRMR TRLasso DKN ViT
Builds on: varying coefficient models linear attention Vision Transformer rotation correlation simplified transformer
Closest papers by meaning
- Adaptive Shrinkage Estimation for Personalized Deep Kernel Regression in Modeling Brain Trajectories (ICLR 2025 · similarity 0.82)
- The Effect of Intrinsic Dataset Properties on Generalization: Unraveling Learning Differences Between Natural and Medical Images (ICLR 2024 · similarity 0.82)
- Beyond Average: Individualized Visual Scanpath Prediction (CVPR 2024 · similarity 0.82)
- Perturbating, Tuning, and Collaborating: Harnessing Vision Foundation Models for Single Domain Generalization on Medical Imaging (AAAI 2025 · similarity 0.82)
- A Statistical Theory of Gated Attention through the Lens of Hierarchical Mixture of Experts (NeurIPS 2026 · similarity 0.82)
- Hierarchical Self-Attention: Generalizing Neural Attention Mechanics to Multi-Scale Problems (NeurIPS 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.