Research map / Architectures and efficiency / Attention and transformers
ChordMixer: A Scalable Neural Attention Model for Sequences with Different Length
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace scaled dot-product attention with ChordMixer (parameter-free rotation plus MLP mixing) | Adding problem with variable lengths · Adding problem, lambda=128k | accuracy | Helped |
| Use ChordMixer instead of padded/truncated Transformer-style attention | Long document classification · Long document classification, all length percentiles | accuracy | Helped |
| Use ChordMixer without padding or chunking | DNA taxonomy classification (Sus vs. Bos) · Genbank Sus vs. Bos, longest percentiles | ROC-AUC | Helped |
| Add self-supervised masked-base pre-training to ChordMixer encoder (ablation) | Genetic variant classification · Genetic Variant Classification | ROC-AUC: +3.2 points (84.9 to 88.1) | Helped |
| Use ChordMixer as classifier instead of Enformer | Genetic variant classification · Genetic Variant Classification | ROC-AUC: +14.4 points (70.5 to 84.9) | Helped |
| Ablation: swap pooling/classifier head (CLS vs AVG) for ChordMixer on Carassius vs. Labeo (ablation) | ChordMixer · Genbank Carassius vs. Labeo | ROC-AUC | No clear effect |
| Use FLAT classifier head instead of AVG/CLS for Poolformer and Cosformer (ablation) | Poolformer, Cosformer · Genbank Carassius vs. Labeo | ROC-AUC | Mixed |
| Replace PoolFormer with ChordMixer-style rotation as token mixer on LRA | LRA ListOps/Text/Image/Retrieval/Pathfinder · LRA benchmark | accuracy | Helped |
Closest papers by meaning
- Posterior Attention Models for Sequence to Sequence Learning (ICLR 2019 · similarity 0.83)
- Scaling Stick-Breaking Attention: An Efficient Implementation and In-depth Study (ICLR 2025 · similarity 0.83)
- FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference (ICLR 2025 · similarity 0.82)
- Tensor Product Attention Is All You Need (NeurIPS 2025 · similarity 0.82)
- ZETA: Leveraging $Z$-order Curves for Efficient Top-$k$ Attention (ICLR 2025 · similarity 0.82)
- Curse of High Dimensionality Issue in Transformer for Long Context Modeling (ICML 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.