Research map / Architectures and efficiency / Attention and transformers
Associative Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Associative Transformer (AiT), a sparse attention architecture whose Global Workspace Layer combines low-rank learnable explicit memory priors, top-k bottleneck attention with a balance loss, and Hopfield-network associative token reconstruction.
Problem
Sparse attention methods such as the Coordination method are parameter-inefficient and fail on complex relational reasoning, motivating a more efficient way to associate sparsely attended tokens.
Approach
Replace iterative cross-attention bottlenecks with low-rank explicit memory whose priors are specialized via top-k bottleneck attention (with a balance loss), then reconstruct tokens by retrieval from the priors as attractors in a continuous Hopfield network.
Key results
- AiT-Base reaches 81.95% average accuracy across CIFAR10, CIFAR100 and Triangle with 91.0M parameters, versus 80.46% for ViT-Base with 85.7M.
- AiT-Medium (6 layers, 45.9M params) averages 81.58%, surpassing ViT-Base while using roughly half the parameters and FLOPs.
- On Sort-of-CLEVR, AiT-Base achieves 80.03% relational accuracy versus 73.43% for Coordination and 63.35% for ViT-Base.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the global workspace layer with explicit memory, bottleneck attention, and Hopfield retrieval | ViT-Base · CIFAR10 test | accuracy: 83.82 ± 0.17 -> 85.44 ± 0.31 | Helped |
| Add the global workspace layer with explicit memory, bottleneck attention, and Hopfield retrieval | ViT-Base · CIFAR100 test | accuracy: 57.92 ± 0.40 -> 60.78 ± 0.25 | Helped |
| Add the global workspace layer with explicit memory, bottleneck attention, and Hopfield retrieval | ViT-Base · Triangle test | accuracy: 99.63 ± 0.15 -> 99.64 ± 0.14 | No clear effect |
| Use AiT-Medium instead of ViT-Base | ImageNet100 · ImageNet100 test | accuracy: 34.62 -> 36.72 | Helped |
| Use AiT-Base instead of Coordination | Sort-of-CLEVR · Sort-of-CLEVR relational | accuracy: 73.43 -> 80.03 | Helped |
| Reset explicit-memory priors every epoch (ablation) | AiT-Small · CIFAR10 test | accuracy: 83.34 ± 0.44 -> 81.94 ± 0.41 | Hurt |
| Remove bottleneck attention balance loss (ablation) | AiT-Small · CIFAR100 test | accuracy: 56.30 ± 0.38 -> 54.72 ± 0.33 | Hurt |
| Replace Hopfield retrieval with cross-attention (ablation) | AiT-Small · CIFAR10 test | accuracy: 83.34 ± 0.44 -> 81.03 ± 0.45 | Hurt |
| Remove the global workspace layer (ablation) | AiT-Small · CIFAR10 test | accuracy: 83.34 ± 0.44 -> 79.53 ± 0.36 | Hurt |
| Replace top-k bottleneck attention with dense pairwise attention (ablation) | AiT-Small · CIFAR100 test | accuracy: 56.30 ± 0.38 -> 46.53 ± 0.41 | Hurt |
| Remove self-attention (ablation) | AiT-Small · CIFAR10 test | accuracy: 83.34 ± 0.44 -> 72.72 ± 0.57 | Hurt |
| Initialize explicit-memory priors uniformly instead of with a Gaussian distribution (ablation) | AiT-Small · CIFAR10 test | accuracy: 83.34 ± 0.44 -> 81.92 ± 0.17 | Hurt |
Surprising findings
- Coordination's accuracy does not improve, and can degrade, as its parameter count and FLOPs increase (Coordination-D and -DH underperform the base Coordination on CIFAR10/Triangle).
- Gaussian prior initialization outperforms the uniform and identity initializations used in prior work.
Limitations
- Scaling to larger datasets requires re-tuning Hopfield inverse temperature beta, which is not adaptive.
- Evaluated only on small and medium vision tasks trained from scratch; more complex reasoning benchmarks (CLEVR, bAbI) are future work.
- Potential vulnerability to adversarial Trojan samples exploiting the sparse bottleneck.
Experimental setup
Datasets: Triangle CIFAR10 CIFAR100 Oxford Pet ImageNet100 Sort-of-CLEVR
Benchmarks: CIFAR10 CIFAR100 Triangle Oxford Pet ImageNet100 Sort-of-CLEVR
Metrics: test accuracy model size (M parameters) FLOPs
Baselines: Coordination Perceiver Set Transformer BRIMs Luna ViT CNN+LSTM Dense networks
Builds on: Coordination Hopfield networks Perceiver Vision Transformer Set Transformer
Closest papers by meaning
- Transformers Learn Faster with Semantic Focus (NeurIPS 2025 · similarity 0.87)
- Disentangling and Integrating Relational and Sensory Information in Transformer Architectures (ICML 2025 · similarity 0.87)
- Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction (ICLR 2025 · similarity 0.85)
- Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers (Student Abstract) (AAAI 2024 · similarity 0.85)
- Task-Free Dynamic Sparse Vision Transformer for Continual Learning (AAAI 2024 · similarity 0.85)
- Understanding Factual Recall in Transformers via Associative Memories (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.