Research map / Architectures and efficiency / Attention and transformers
Sparse Sinkhorn Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace standard attention with learned block sorting and local Sinkhorn attention | Transformer · Algorithmic sorting, length 512 | EM: 45.69 -> 49.24 | Helped |
| Use Sinkhorn attention instead of standard attention | Transformer Base · LM1B subword | Perplexity: 41.57 -> 40.79 | Helped |
| Use Sinkhorn attention instead of standard attention | Transformer Big · LM1B subword | Perplexity: 27.59 -> 28.39 | Hurt |
| Mix Sinkhorn attention with vanilla attention | Sinkhorn Transformer · LM1B subword, Base and Big | Perplexity: 40.79 -> 40.11; 28.39 -> 27.34 | Helped |
| Use Sinkhorn attention instead of standard attention | Transformer · LM1B character, Base and Big | Bytes per char (Bpc): 1.283 -> 1.295; 1.121 -> 1.132 | Hurt |
| Mix Sinkhorn attention with vanilla attention | Sinkhorn Transformer · LM1B character, Base and Big | Bytes per char (Bpc): 1.295 -> 1.270; 1.132 -> 1.119 | Helped |
| Use Sinkhorn attention instead of standard attention | Transformer · CIFAR-10 pixel-wise image generation | Bpd: 3.198 -> 3.197 | Helped |
| Use SortCut truncation with budget 2x16 | Sinkhorn Transformer · SNLI | Accuracy: 78.87 -> 80.30 | Helped |
| Use SortCut truncation with budget 2x32 | Sinkhorn Transformer · MNLI | Accuracy: 53.69 -> 55.80 | Helped |
| Use a single linear layer instead of the two-layer SortNet with output ReLU (ablation) | Sparse Sinkhorn Attention · LM1B subword, Base, b=32 | Perplexity: 41.70 -> 41.29 | Helped |
| Tie the key and value weights (ablation) | Sparse Sinkhorn Attention · LM1B subword, Base, b=32 | Perplexity: 41.29 -> 42.26 | Hurt |
| Remove Sinkhorn normalization (ablation) | Sparse Sinkhorn Attention · LM1B subword, Base, b=32 | Perplexity: 41.29 -> 52.40 | Hurt |
Closest papers by meaning
- Delta Attention: Fast and Accurate Sparse Attention Inference by Delta Correction (NeurIPS 2025 · similarity 0.89)
- More Value per Key: Asymmetric Sparse Attention for Faster LLM Decoding (NeurIPS 2026 · similarity 0.88)
- TidalDecode: Fast and Accurate LLM Decoding with Position Persistent Sparse Attention (ICLR 2025 · similarity 0.87)
- SEA: Sparse Linear Attention with Estimated Attention Mask (ICLR 2024 · similarity 0.87)
- Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps (NeurIPS 2026 · similarity 0.86)
- Scaling Stick-Breaking Attention: An Efficient Implementation and In-depth Study (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.