Research map / Architectures and efficiency / Attention and transformers
Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add encoder self-attention to an rFF encoder (ablation) | Set Transformer · CIFAR-100 clustering | ARI0: 0.7612 $\pm$ 0.0237 -> 0.9210 $\pm$ 0.0055 | Helped |
| Use PMA instead of pooling (ablation) | SAB encoder · Unique character counting | accuracy: 0.5659 $\pm$ 0.0067 -> 0.6037 $\pm$ 0.0072 | Helped |
| Use 2, 4, or 8 PMA seed vectors instead of 1 (ablation) | SAB + PMA · Unique character counting | accuracy: 0.6037 $\pm$ 0.0072 -> 0.5806 $\pm$ 0.0075; 0.5945 $\pm$ 0.0072; 0.6001 $\pm$ 0.0078 | Hurt |
| Use PMA instead of pooling (ablation) | ISAB (32) encoder · 2D synthetic Gaussian mixtures | LL0/data: -1.6353 $\pm$ 0.0182 -> -1.4963 $\pm$ 0.0064 | Helped |
| Use 32 inducing points rather than 16 or 64 (ablation) | ISAB + PMA · 2D synthetic Gaussian mixtures | LL0/data: -1.5009 $\pm$ 0.0068; -1.4963 $\pm$ 0.0064; -1.5042 $\pm$ 0.0158 | Mixed |
| Add self-attention to an rFF encoder (ablation) | PMA-based Set Transformer · Large-scale 2D synthetic Gaussian mixtures | LL0/data: -2.0515 $\pm$ 0.0067 -> -1.8928 $\pm$ 0.0076 | Helped |
| Use ISAB with 16 inducing points rather than SAB (ablation) | PMA-based Set Transformer · CIFAR-100 clustering | ARI0: 0.9015 $\pm$ 0.0097 -> 0.9210 $\pm$ 0.0055 | Helped |
| Replace PMA with pooling (ablation) | SAB encoder · CIFAR-100 clustering | ARI0: 0.9015 $\pm$ 0.0097 -> 0.5831 $\pm$ 0.0341 | Hurt |
Closest papers by meaning
- Learning Representations of Sets through Optimized Permutations (ICLR 2019 · similarity 0.86)
- Set Norm and Equivariant Skip Connections: Putting the Deep in Deep Sets (ICML 2022 · similarity 0.86)
- SeTformer Is What You Need for Vision and Language (AAAI 2024 · similarity 0.86)
- Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction (ICLR 2025 · similarity 0.85)
- Learning Linear Attention in Polynomial Time (NeurIPS 2025 · similarity 0.85)
- Hierarchical Self-Attention: Generalizing Neural Attention Mechanics to Multi-Scale Problems (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.