Research map / Architectures and efficiency / Attention and transformers
SeTformer Is What You Need for Vision and Language
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Replaces dot-product self-attention in transformers with Self-optimal Transport (SeT), an entropic optimal-transport alignment computed in an RKHS against K-means reference centroids, yielding SeTformer for vision and language tasks.
Problem
Dot-product self-attention has quadratic time and memory cost in sequence length, and existing kernel-based linear attention approximations often lose accuracy relative to softmax attention.
Approach
Replace the softmax attention matrix with alignment weights from entropic optimal transport between input features and a small set of K-means reference centroids, embedded in an RKHS via Nystrom approximation, with linear positional encoding; this keeps non-negativity and nonlinear reweighting while reducing cost.
Key results
- SeTformer-B reaches 86.2% top-1 on ImageNet-1K at 224x224 with 56.7M params and 10.2G FLOPs, versus NAT-B 84.4%.
- SeTformer-B achieves 51.9 box AP on COCO Mask R-CNN 3x, +1.1 over CSWin-B (50.8) with fewer params and FLOPs.
- SeTformer-T reaches 83.9% top-1 on ImageNet-1K, surpassing CSWin-T by +1.2% and FocalNet-T by +1.6%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace softmax self-attention with SeT | SeTformer-T · ImageNet-1K 224×224 | top-1 accuracy: 83.2 -> 83.9 | Helped |
| replace softmax self-attention with SeT | SeTformer-B · ImageNet-1K 224×224 | top-1 accuracy: 84.4 -> 86.2 | Helped |
| use linear positional encoding instead of no positional encoding (ablation) | SeTformer · ImageNet-1K | top-1 accuracy: 83.78 -> 84.72 | Helped |
| use linear positional encoding instead of no positional encoding (ablation) | SeTformer · COCO | box AP: 49.63 -> 51.32 | Helped |
| use absolute positional encoding instead of no positional encoding (ablation) | SeTformer · ImageNet-1K | top-1 accuracy: 83.78 -> 83.96 | Helped |
| use relative positional encoding instead of no positional encoding (ablation) | SeTformer · ImageNet-1K | top-1 accuracy: 83.78 -> 84.35 | Helped |
| replace OT with dot product (ablation) | SeTformer · GLUE QQP | accuracy: 89.36 -> 87.85 | Hurt |
| replace OT with Sliced-OT (ablation) | SeTformer · GLUE QQP | accuracy: 89.36 -> 86.74 | Hurt |
| add a reference set (ablation) | SeTformer · GLUE QQP | accuracy: 88.92 -> 89.36 | Helped |
| remove positional encoding (ablation) | SeTformer · GLUE SST-2 | accuracy: 93.54 -> 88.25 | Hurt |
| increase reference count from m=100 to m=500 (ablation) | SeTformer-T · ImageNet | top-1 accuracy: 79.81 -> 83.91 | Helped |
| increase reference count from m=500 to m=1000 (ablation) | SeTformer-T · ImageNet | top-1 accuracy: 83.91 -> 81.74 | Hurt |
Surprising findings
- Using a dot product instead of OT in the same architecture reduces GLUE QQP accuracy from 89.36 to 87.85 in the ablation.
- Performer-style kernel attention drops ImageNet top-1 by about 2 points relative to the vanilla T2T-ViT baseline.
Limitations
- Performance is sensitive to the number of reference points m, which requires careful tuning per task.
- Reported gains on ImageNet depend on hyperparameters (epsilon, tau, m) tuned across tasks.
Experimental setup
Datasets: ImageNet-1K COCO ADE20K WikiText-103 GLUE Amazon
Benchmarks: ImageNet-1K COCO ADE20K GLUE Amazon
Metrics: top-1 accuracy AP box/mask mIoU accuracy FLOPs throughput
Baselines: Swin ConvNeXt FocalNet CSWin MViTv2 NAT Performer Reformer ScatterBrain KDEformer Longformer RoBERTa
Builds on: Swin Transformer Nystrom approximation Performer FocalNet NAT
Closest papers by meaning
- Long-Short Transformer: Efficient Transformers for Language and Vision (NeurIPS 2021 · similarity 0.88)
- Theory, Analysis, and Best Practices for Sigmoid Self-Attention (ICLR 2025 · similarity 0.86)
- MemoryFormer : Minimize Transformer Computation by Removing Fully-Connected Layers (NeurIPS 2024 · similarity 0.86)
- PADRe: A Unifying Polynomial Attention Drop-in Replacement for Efficient Vision Transformer (ICLR 2025 · similarity 0.85)
- Rethinking Decoders for Transformer-based Semantic Segmentation: A Compression Perspective (NeurIPS 2024 · similarity 0.85)
- Bridging the Divide: Reconsidering Softmax and Linear Attention (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.