Research map / Architectures and efficiency / Attention and transformers

Elliptical Attention

NeurIPS 2024 · Architectures and efficiency › Attention and transformers · approach: Attention head and block redesign · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Elliptical Attention, which replaces the Euclidean distance in dot-product self-attention with a Mahalanobis metric whose diagonal weights come from a parameter-free estimate of coordinate-wise variability, to reduce representation collapse and improve robustness.

Problem

Standard dot-product self-attention uses Euclidean (spherically invariant) neighborhoods, which the authors argue causes high-variance estimates, representation collapse, and vulnerability to contaminated inputs.

Approach

Interpret self-attention as Nadaraya-Watson regression with isotropic Gaussian kernels and replace the isotropic kernel with a Mahalanobis (diagonal-M) norm, where M is estimated from layer-to-layer differences of value vectors, stretching the feature space toward directions of high contextual relevance without learnable parameters.

Key results

• On WikiText-103 with 2.5% word-swap contamination, Transformer-Elliptical reaches 32.00 clean test PPL (best) vs 34.29 for the Transformer baseline, and 52.59 contaminated PPL (second best).
• On ImageNet-1K under SPSA black-box attack, DeiT-Elliptical reaches 56.55 top-1 vs 48.34 for DeiT, and 44.96 top-1 under PGD.
• Elliptical achieves top average LRA accuracy of 59.4 versus 59.0 for Longformer and 58.5 for the vanilla Transformer.

Measured effects reported in the paper

Surprising findings

• DeiT-Elliptical is slightly below the DeiT baseline on clean ImageNet top-1 for the tiny setting (72.36 vs 72.23 is actually higher; MoM and KDE variants are lower), so gains are mostly under attack rather than clean accuracy in the vision setting.
• On contaminated WikiText-103 the MoM baseline achieves lower contaminated PPL (52.14) than Elliptical (52.59).

Limitations

• Coordinate-wise variability is estimated from differences of values across neighboring layers, which the authors note is prone to noise.
• Theoretical guarantees rely on assumptions (e.g., attenuation and near-equal expectations) that are only approximate in places.

Experimental setup

Datasets: WikiText-103 ImageNet-1K ADE20K Long Range Arena (ListOps, Text, Retrieval, Image, Pathfinder)

Benchmarks: WikiText-103 ImageNet-1K Long Range Arena ADE20K

Metrics: perplexity top-1 accuracy top-5 accuracy pixel accuracy mean IOU average accuracy

Baselines: Transformer Performer Transformer-MGK FourierFormer Transformer-SPKDE Transformer-MoM DeiT Distill RVT DeiT-KDE FAN Linformer

Builds on: Nadaraya-Watson regression transformer Performer FourierFormer Mixture of Gaussian Keys

Closest papers by meaning

• Scale-invariant attention (NeurIPS 2025 · similarity 0.86)
• Theory, Analysis, and Best Practices for Sigmoid Self-Attention (ICLR 2025 · similarity 0.86)
• Mixture-of-Top-$k$ Attention: Efficient Attention as Scalable Fast Weights (NeurIPS 2026 · similarity 0.85)
• Dissecting Query-Key Interaction in Vision Transformers (NeurIPS 2024 · similarity 0.85)
• Bridging the Divide: Reconsidering Softmax and Linear Attention (NeurIPS 2024 · similarity 0.85)
• Data-Adaptive Mahalanobis Metric Learning for Cross-Head Attention in Transformers (NeurIPS 2026 · similarity 0.84)