Research map / Architectures and efficiency / Attention and transformers
Multiple Positional Self-Attention Network for Text Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Backward and Forward masks (ablation) | MPSAN · SST test | accuracy (%): 49.19 -> 51.54 | Helped |
| add Faraway masks FA(2) and FA(3) (ablation) | MPSAN · SST test | accuracy (%): 51.54 -> 51.99 | Helped |
| add Scaled-Distance mask on BW and FW (ablation) | MPSAN · SST test | accuracy (%): 51.99 -> 52.53 | Helped |
| use Scaled-Distance instead of Distance mask on BW/FW (ablation) | MPSAN · SST test | accuracy (%): 52.17 -> 52.53 | Helped |
| also add Scaled-Distance mask on the Faraway masks (ablation) | MPSAN · SST test | accuracy (%): 52.53 -> 52.48 | Hurt |
| replace Fusion Gate with Tokens-Position-Weight (ablation) | MPSAN · SST test | accuracy (%): 52.39 -> 52.53 | Helped |
| replace Fusion Gate with Tokens-Position-Weight (ablation) | MPSAN · SST test | parameters (M): 4.33m -> 1.09m | Helped |
| use multi-dimensional instead of 1-D self-attention (ablation) | MPSAN · SST test | accuracy (%): 52.53 -> 52.67 | No clear effect |
| add Position-Fusion Layer (ablation) | MPSAN · SST test | accuracy (%): 51.22 -> 52.53 | Helped |
| add Positional Self-Attention Layer plus fusion (ablation) | MPSAN · SST test | accuracy (%): 49.19 -> 52.53 | Helped |
| add attention modules (ablation) | MPSAN · SST test | accuracy (%): 44.93 -> 52.53 | Helped |
| MPSAN vs DiSAN | MPSAN · SICK | Pearson's r: 0.8695 -> 0.8674 | Hurt |
Closest papers by meaning
- DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding (AAAI 2018 · similarity 0.87)
- A STRUCTURED SELF-ATTENTIVE SENTENCE EMBEDDING (ICLR 2017 · similarity 0.86)
- SparseBERT: Rethinking the Importance Analysis in Self-attention (ICML 2021 · similarity 0.86)
- Rethinking Positional Encoding in Language Pre-training (ICLR 2021 · similarity 0.84)
- Selective Attention: Enhancing Transformer through Principled Context Control (NeurIPS 2024 · similarity 0.83)
- Attention with Trained Embeddings Provably Selects Important Tokens (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.