Research map / Architectures and efficiency / Attention and transformers
Token-Context Attention for NLI: An Alternative to Self-Attention
Xin Zhang, Victor S. Sheng
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Token-Context Attention with self-attention (ablation) | T-C Attn · ANLI R1 | accuracy (%): 35.95 -> 38.68 | Improved |
| enable backpropagation through real part (ablation) | T-C Attn · ANLI R1 | accuracy (%): 33.57 -> 38.68 | Improved |
| enable backpropagation through imaginary part (ablation) | T-C Attn · ANLI R1 | accuracy (%) | No clear effect |
| add dual directional gates (ablation) | T-C Attn · ANLI R1 | accuracy (%): 38.64 -> 38.68 | No clear effect |
| add dual directional gates (ablation) | T-C Attn · ANLI R2 (4 hard seeds) | accuracy (%): 42.10 -> 43.15 | Improved |
| add dual directional gates (ablation) | T-C Attn · ANLI R3 (2 hard seeds) | accuracy (%): 40.42 -> 41.63 | Improved |
| learned global semantic vector instead of fixed vector 1, or random phase (ablation) | T-C Attn · ANLI R1 | accuracy (%): approx 33 -> 38.68 | Improved |
| T-C Attn vs Phi 1.5 (0.8B) APP | T-C Attn · ANLI R3 | APP: 48.55% -> 50.20% | Mixed |
| T-C Attn vs Mamba (0.8B) APP | T-C Attn · ANLI R3 | APP: 51.44% -> 50.20% | Hurt |
| T-C Attn vs Llama3.2 (0.85B) APP | T-C Attn · ANLI R1 | APP: 42.36% -> 46.60% | Improved |
| T-C Attn vs 1B self-attention models, FGSM and single-step PGD robustness | T-C Attn · ANLI R1 attacks | accuracy drop | Improved |
Closest papers by meaning
- OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference (ICML 2025 · similarity 0.88)
- Rodimus*: Breaking the Accuracy-Efficiency Trade-Off with Efficient Attentions (ICLR 2025 · similarity 0.87)
- Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration (ICML 2024 · similarity 0.87)
- Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference (AAAI 2025 · similarity 0.87)
- ReAttention: Training-Free Infinite Context with Finite Attention Scope (ICLR 2025 · similarity 0.86)
- RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
