AI Paper Atlas

Research map / Adversarial robustness and security / Certified and empirical robustness

Designing Robust Transformers using Robust Kernel Density Estimation

Xing Han, Tongzheng Ren, Tan Minh Nguyen, Khai Nguyen, Joydeep Ghosh, Nhat Ho

NeurIPS 2023 · Adversarial robustness and security › Certified and empirical robustness · approach: Adversarial training with feature consistency · 2 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace softmax attention with RKDE (Huber) attentionTransformer · WikiText-103 clean (small)test perplexity: 34.29 -> 32.29Improved
replace softmax attention with SPKDE attentionTransformer · WikiText-103 word swap (small)test perplexity: 74.56 -> 54.97Improved
replace softmax attention with MoM attentionTransformer · WikiText-103 clean (small)test perplexity: 34.29 -> 34.68Hurt
replace softmax attention with MoM attentionTransformer · WikiText-103 word swap (small)test perplexity: 74.56 -> 52.14Improved
add MoM to FourierFormerFourierFormer · WikiText-103 word swap (small)test perplexity: 68.33 -> 50.96Improved
replace softmax with SPKDE attentionViT · ImageNet PGDtop-1 accuracy: 41.84 -> 44.51Improved
replace softmax with SPKDE attentionViT · ImageNet cleantop-1 accuracy: 72.23 -> 73.22Improved
replace softmax with MoM attentionViT · ImageNet cleantop-1 accuracy: 72.23 -> 71.94Hurt
replace softmax with SPKDE attentionViT · ImageNet-OAUPR: 18.31 -> 28.42Improved
replace softmax with MoM attentionViT · ImageNet-Atop-1 accuracy: 0.18 -> 2.29Improved
replace softmax with SPKDE attentionTransformer · UEA UWaveaccuracy: 84.45 -> 88.14Improved
replace softmax with MoM attentionTransformer · UEA Spoken Arabicaccuracy: 99.33 -> 97.64Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy