Research map / Architectures and efficiency / Attention and transformers
Continuous Self-Attention Models with Neural ODE Networks
Jing Zhang, Peng Zhang, Baiwen Kong, Junqiu Wei, Xin Jiang
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CSAODE instead of MPSAN | text classification models · CR test | accuracy: 85.4 -> 87 | Improved |
| use CSAODE instead of MPSAN | text classification models · MPQA test | accuracy: 90.4 -> 90.8 | Improved |
| use CSAODE instead of MPSAN | text classification models · SUBJ test | accuracy: 94.6 -> 95.9 | Improved |
| use CSAODE instead of MPSAN | text classification models · TREC test | accuracy: 94.8 -> 95.4 | Improved |
| use CSAODE instead of Star-Transformer | SNLI · SNLI test | accuracy: 86.0 -> 86.1 | Improved |
| use CSAODE instead of CNN+Transformer | WikiQA · WikiQA test | MRR: 0.703 -> 0.706 | Improved |
| remove ARKNet and use one self-attention block (ablation) | CSAARK · TREC test | accuracy: 95.2 -> 93.7 | Hurt |
| replace the ODE solver with five residual-connected self-attention blocks (ablation) | CSAARK · TREC test | accuracy: 95.2 -> 94.2 | Hurt |
| add N-gram convolution to capture local information (ablation) | CSAARK · TREC test | accuracy: 92.8 -> 95.2 | Improved |
| reduce hidden dimension from 300 to 50 (ablation) | CSAARK · TREC test | accuracy: 95.2 -> 91.6 | Hurt |
| increase hidden dimension from 200 to 300 (ablation) | CSAARK · TREC test | accuracy: 94.3 -> 95.2 | Improved |
| increase convolution window size from 1 to 2 (ablation) | CSAARK · TREC test | accuracy: 94.1 -> 95.2 | Improved |
Closest papers by meaning
- On the Dynamics of Training Attention Models (ICLR 2021 · similarity 0.87)
- Untangling tradeoffs between recurrence and self-attention in artificial neural networks (NeurIPS 2020 · similarity 0.86)
- Understanding the Differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networks (NeurIPS 2024 · similarity 0.85)
- From Self-Attention to Markov Models: Unveiling the Dynamics of Generative Transformers (ICML 2024 · similarity 0.85)
- Training Dynamics of In-Context Learning in Linear Attention (ICML 2025 · similarity 0.85)
- SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
