Research map / Architectures and efficiency / Attention and transformers
Hyper-SET: Designing Transformers via Hyperspherical Energy Minimization
Yunzhe Hu, Difan Zou, Dong Xu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use bi-softmax attention instead of sigmoid attention (ablation) | Hyper-SET · CIFAR-10 | top-1 accuracy (%): 85.93 -> 90.11 | Improved |
| use bi-softmax attention instead of linear attention (ablation) | Hyper-SET · CIFAR-100 | top-1 accuracy (%): 56.97 -> 63.41 | Improved |
| use ReLU feedforward instead of softmax feedforward (ablation) | Hyper-SET · CIFAR-10 | top-1 accuracy (%): 88.20 -> 90.11 | Improved |
| use ReLU feedforward instead of gated feedforward (ablation) | Hyper-SET · CIFAR-100 | top-1 accuracy (%): 59.29 -> 63.41 | Improved |
| learn step sizes instead of fixing them to 0.5 (ablation) | Hyper-SET · CIFAR-10 | top-1 accuracy (%): 25.81 -> 90.11 | Improved |
| learn step sizes instead of fixing them to 0.1 (ablation) | Hyper-SET · CIFAR-10 | top-1 accuracy (%): 81.45 -> 90.11 | Improved |
| add depth-wise LoRA rank 4 (ablation) | Hyper-SET · ImageNet-100 | top-1 accuracy (%): 70.16 -> 70.36 | Improved |
| add depth-wise LoRA rank 32 (ablation) | Hyper-SET · ImageNet-100 | top-1 accuracy (%): 70.16 -> 72.20 | Improved |
| scale up hidden dimension to match Transformer parameters | Hyper-SET · ImageNet-100 | top-1 accuracy (%): 67.64 -> 70.16 | Improved |
| scale up iterations to 24 and feedforward width to 8d | Hyper-SET · ImageNet-100 masked image modeling (40% mask) | FID: 59.841 -> 45.174 | Improved |
| use Hyper-SET instead of Transformer with fewer parameters | Sudoku · Sudoku | accuracy (%): 49.30 -> 54.70 | Improved |
| use Hyper-SET Small instead of Transformer on CIFAR-10 | Image classification · CIFAR-10 | top-1 accuracy (%): 89.90 -> 88.89 | Hurt |
Closest papers by meaning
- nGPT: Normalized Transformer with Representation Learning on the Hypersphere (ICLR 2025 · similarity 0.87)
- Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding (NeurIPS 2025 · similarity 0.86)
- TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters (ICLR 2025 · similarity 0.86)
- Scaling Diffusion Transformers Efficiently via $\mu$P (NeurIPS 2025 · similarity 0.86)
- FrameQuant: Flexible Low-Bit Quantization for Transformers (ICML 2024 · similarity 0.85)
- Viewing Transformers Through the Lens of Long Convolutions Layers (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
