Research map / Adversarial robustness and security / Certified and empirical robustness
Designing Robust Transformers using Robust Kernel Density Estimation
Xing Han, Tongzheng Ren, Tan Minh Nguyen, Khai Nguyen, Joydeep Ghosh, Nhat Ho
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace softmax attention with RKDE (Huber) attention | Transformer · WikiText-103 clean (small) | test perplexity: 34.29 -> 32.29 | Improved |
| replace softmax attention with SPKDE attention | Transformer · WikiText-103 word swap (small) | test perplexity: 74.56 -> 54.97 | Improved |
| replace softmax attention with MoM attention | Transformer · WikiText-103 clean (small) | test perplexity: 34.29 -> 34.68 | Hurt |
| replace softmax attention with MoM attention | Transformer · WikiText-103 word swap (small) | test perplexity: 74.56 -> 52.14 | Improved |
| add MoM to FourierFormer | FourierFormer · WikiText-103 word swap (small) | test perplexity: 68.33 -> 50.96 | Improved |
| replace softmax with SPKDE attention | ViT · ImageNet PGD | top-1 accuracy: 41.84 -> 44.51 | Improved |
| replace softmax with SPKDE attention | ViT · ImageNet clean | top-1 accuracy: 72.23 -> 73.22 | Improved |
| replace softmax with MoM attention | ViT · ImageNet clean | top-1 accuracy: 72.23 -> 71.94 | Hurt |
| replace softmax with SPKDE attention | ViT · ImageNet-O | AUPR: 18.31 -> 28.42 | Improved |
| replace softmax with MoM attention | ViT · ImageNet-A | top-1 accuracy: 0.18 -> 2.29 | Improved |
| replace softmax with SPKDE attention | Transformer · UEA UWave | accuracy: 84.45 -> 88.14 | Improved |
| replace softmax with MoM attention | Transformer · UEA Spoken Arabic | accuracy: 99.33 -> 97.64 | Hurt |
Closest papers by meaning
- ProTransformer: Robustify Transformers via Plug-and-Play Paradigm (NeurIPS 2024 · similarity 0.86)
- Sparse Attention as Compact Kernel Regression (NeurIPS 2026 · similarity 0.86)
- Transformer Meets Twicing: Harnessing Unattended Residual Information (ICLR 2025 · similarity 0.85)
- DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averaging (NeurIPS 2024 · similarity 0.85)
- Transformers Learn Low Sensitivity Functions: Investigations and Implications (ICLR 2025 · similarity 0.85)
- Understanding Differential Transformer Unchains Pretrained Self-Attentions (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
