AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

FourierFormer: Transformer Meets Generalized Fourier Integral Theorem

Tan Minh Nguyen, Minh Pham, Tam Minh Nguyen, Khai Nguyen, Stanley Osher, Nhat Ho

NeurIPS 2022 · Architectures and efficiency › Attention and transformers · approach: Attention head and block redesign · 3 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace dot-product attention with Fourier attention (small)Transformer LM · WikiText-103 testperplexity: 34.29 -> 32.85Improved
replace dot-product attention with Fourier attention (medium)Transformer LM · WikiText-103 testperplexity: 29.60 -> 28.01Improved
increase model size from small to mediumFourierFormer vs baseline · WikiText-103 validperplexity gain: 1.29 -> 1.39Improved
replace dot-product attention with Fourier attentionDeiT · ImageNettop-1 accuracy: 72.23 -> 73.25Improved
replace dot-product attention with Fourier attentionDeiT · ImageNettop-5 accuracy: 91.13 -> 91.66Improved
replace softmax attention with Fourier attentionTransformer · UEA averageaccuracy: 72.07 -> 73.17Improved
replace softmax attention with Fourier attentionTransformer · UEA PEMS-SFaccuracy: 82.66 -> 86.70Improved
replace softmax attention with Fourier attentionTransformer · UEA SpokenArabicDigitsaccuracy: 99.33 -> 99.00Hurt
replace softmax attention with Fourier attentionTransformer · UEA Handwritingaccuracy: 32.08 -> 31.68Hurt
replace softmax attention with Fourier attentionDecision Transformer · D4RL averagereward: 74.39 -> 75.69Improved
replace softmax attention with Fourier attentionDecision Transformer · D4RL medium-replay Walkerreward: 66.11 -> 63.19Hurt
replace softmax attention with Fourier attentionTransformer · IWSLT14 De-EnBLEU: 34.42 -> 34.68Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy