Research map / Architectures and efficiency / Attention and transformers
Energy Transformer
Benjamin Hoover, Yuchen Liang, Bao Pham, Rameswar Panda, Hendrik Strobelt, Duen Horng Chau, Mohammed J Zaki, Dmitry Krotov
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Energy Transformer instead of graph anomaly baselines | Graph anomaly detection · Yelp, Amazon, T-Finance, T-Social; 1% and 40% training | Macro-F1 | Mixed |
| Use Energy Transformer instead of graph anomaly baselines | Graph anomaly detection · Yelp, Amazon, T-Finance, T-Social; 1% and 40% training | AUC | Mixed |
| Use Energy Transformer graph classifier | Graph classification · TUDataset | accuracy | Mixed |
| Use energy-based token routing to avoid neighborhood averaging | Graph anomaly detection · Graph anomaly detection | Macro-F1 | Improved |
| Use step size 0.1 rather than the traditional step size 1 (ablation) | ImageNet masked image completion ET · ImageNet-1k masked image completion | reconstruction quality | Improved |
| Include some unoccluded patches in the MSE loss (ablation) | ImageNet masked image completion ET · ImageNet-1k masked image completion | HN filter learning | Improved |
| Set attention-energy beta too high (ablation) | ImageNet masked image completion ET · ImageNet-1k masked image completion | training stability | Hurt |
| Remove gradient clipping (ablation) | ImageNet masked image completion ET · ImageNet-1k masked image completion | training stability | Hurt |
| Increase learning rate with gradient clipping (ablation) | ImageNet masked image completion ET · ImageNet-1k masked image completion | training speed | Improved |
Closest papers by meaning
- On the Role of Hidden States of Modern Hopfield Network in Transformer (NeurIPS 2025 · similarity 0.86)
- What Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysis (ICLR 2025 · similarity 0.86)
- Associative Transformer (CVPR 2025 · similarity 0.86)
- Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers (Student Abstract) (AAAI 2024 · similarity 0.85)
- Towards Principled Graph Transformers (NeurIPS 2024 · similarity 0.85)
- Energy-Based Transformers are Scalable Learners and Thinkers (ICLR 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
