Research map / Architectures and efficiency / Attention and transformers
Trading Complexity for Expressivity Through Structured Generalized Linear Token Mixing
Erwan Fagnou, Paul Caillon, Blaise Delattre, Alexandre Allauzen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use cache-efficient variant (ablation) | f(k)=2^k · Copy | accuracy (%): 92.63 -> 75.47 | Hurt |
| use cache-efficient variant (ablation) | f(k)=2^k · Associative recall | accuracy (%): 49.03 -> 52.59 | Improved |
| use cache-efficient variant (ablation) | f(k)=k^2+1 · Copy | accuracy (%): 99.66 -> 91.59 | Hurt |
| use cache-efficient variant (ablation) | f(k)=k^2+1 · Associative recall | accuracy (%): 53.61 -> 54.56 | Improved |
| use dense recurrence instead of attention | token mixer · Multi-hop recall | accuracy (%): 39.21 -> 99.80 | Improved |
| use k-th order recurrence instead of diagonal SSM | recurrence · Copy | accuracy (%): 42.98 -> 74.66 | Improved |
| use f(k)=k^2+1 instead of f(k)=2^k | translation-invariant pattern · Copy | accuracy (%): 92.63 -> 99.66 | Improved |
| use f(k)=k^2+1 instead of f(k)=2^k | translation-invariant pattern · Multi-hop recall | accuracy (%): 34.85 -> 35.68 | No clear effect |
| use Mamba parameterization of A and B instead of the authors' one (ablation) | first-order recurrence · OpenWebText 125M | perplexity: 31.28 -> 30.64 | Improved |
| use local attention instead of dense recurrence | token mixer · Copy | accuracy (%): 100.00 -> 23.75 | Hurt |
Closest papers by meaning
- Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation (NeurIPS 2025 · similarity 0.88)
- Mean-Field Parallel Decoding for Discrete Diffusion Language Models (NeurIPS 2026 · similarity 0.87)
- Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions (ICML 2025 · similarity 0.86)
- Polestar: Drift-Aware Cache Calibration and Token Commitment for Efficient Inference of Diffusion LLMs (NeurIPS 2026 · similarity 0.86)
- MARS: Enabling Autoregressive Models Multi-Token Generation (NeurIPS 2026 · similarity 0.85)
- Orthrus: Memory-Efficient Parallel Token Generation via Dual-View Diffusion (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
