AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Your Transformer May Not be as Powerful as You Expect

Shengjie Luo, Shanda Li, Shuxin Zheng, Tie-Yan Liu, Liwei Wang, Di He

NeurIPS 2022 · Architectures and efficiency › Attention and transformers · approach: Transformer expressivity and learnability · 12 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Add URPE-based Attention to Transformer-XLTransformer-XL Base · WikiText-103 validation and testperplexity: 23.1 -> 22.4; 24.0 -> 23.2Improved
Add URPE-based Attention to GraphormerGraphormer · ZINC-Subset testMAE: 0.122$\pm$0.006 -> 0.086$\pm$0.007Improved
Add URPE-based Attention to GraphormerGraphormer · ZINC-Full testMAE: 0.052$\pm$0.005 -> 0.028$\pm$0.002Improved
Add URPE-based Attention to GraphormerGraphormer · PCQM4M validationMAE: 0.1264 -> 0.1238Improved
Add URPE-based Attention to GraphormerGraphormer · MAG240M validationaccuracy: 0.7013 -> 0.7074Improved
Add URPE-based Attention (ablation)Transformer-XL · WikiText-103 validation, L=4perplexity: 29.6 -> 28.7Improved
Add URPE-based Attention (ablation)Transformer-XL · WikiText-103 validation, L=8perplexity: 26.0 -> 25.2Improved
Add URPE-based Attention (ablation)Transformer-XL · WikiText-103 validation, L=16perplexity: 23.1 -> 22.4Improved
Use URPE-based Attention instead of RPE (ablation)Transformer · Inference, N=128runtime (ms in log base 2): 4.55 -> 4.59Hurt
Use URPE-based Attention instead of RPE (ablation)Transformer · Inference, N=256peak memory (GB): 1.12 -> 1.17Hurt
Use URPE-based Attention instead of RPE (ablation)Transformer · Inference, N=512peak memory (GB): 1.86 -> 2.04Hurt
Use URPE-based Attention instead of T5-style RPEVanilla Transformer · Synthetic Position Identification and Even Token Predictiontoken-level accuracy: less than 60\% -> 100\%Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy