Research map / Architectures and efficiency / Attention and transformers
Compositional Attention: Disentangling Search and Retrieval
Sarthak Mittal, Sharath Chandra Raparthy, Irina Rish, Yoshua Bengio, Guillaume Lajoie
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace rigid search-retrieval head pairings with dynamically composed retrievals | Transformer attention · Sort-of-CLEVR | Binary accuracy: 84.4 ± 5.3 -> 89.8 ± 1.1 | Improved |
| Replace rigid search-retrieval head pairings with dynamically composed retrievals | Transformer attention · Sort-of-CLEVR | Ternary accuracy: 64.9 ± 3.3 -> 67.1 ± 1.5 | Improved |
| Replace multi-head attention with compositional attention | Universal Transformer · Multi-Task Image Classification (CIFAR10) | Accuracy: 77.2 ± 0.3 -> 78.0 ± 0.1 | Improved |
| Replace multi-head attention with compositional attention | Universal Transformer · Multi-Task Image Classification (FashionMNIST) | Accuracy: 89.4 ± 0.0 -> 90.0 ± 0.0 | Improved |
| Replace multi-head attention with compositional attention | Universal Transformer · Multi-Task Image Classification (SVHN) | Accuracy: 85.0 ± 0.1 -> 86.0 ± 0.7 | Improved |
| Replace multi-head attention with compositional attention | Universal Transformer · Multi-Task Image Classification (Equilateral Triangle) | Accuracy: 96.8 ± 0.1 -> 97.2 ± 0.3 | Improved |
| Replace multi-head attention with compositional attention | Transformer · Equilateral Triangle Detection | Test accuracy: 93.8 ± 0.1 -> 97.0 ± 0.3 | Improved |
| Replace multi-head attention with compositional attention | Transformer · WikiText-103 test | Perplexity: 39.6 ± 0.3 -> 38.8 ± 0.0 | Improved |
| Use 1 retrieval instead of 4 (ablation) | Compositional Transformer · Sort-of-CLEVR | Ternary accuracy: 64.8 ± 4.1 -> 66.4 ± 1.3 | Improved |
| Use 4 retrievals instead of 3 (ablation) | Compositional Transformer · Sort-of-CLEVR | Ternary accuracy: 67.1 ± 1.5 -> 64.8 ± 4.1 | Hurt |
| Use 1 retrieval instead of 4 (ablation) | Compositional Transformer · Multi-Task Image Classification (CIFAR10) | Accuracy: 77.2 ± 0.3 -> 77.5 ± 0.5 | No clear effect |
| Replace dot-product retrieval selection with a linear MLP (ablation) | Compositional attention · Sort-of-CLEVR and Equilateral Triangle Detection | Performance | Mixed |
Closest papers by meaning
- Attention as a Hypernetwork (ICLR 2025 · similarity 0.89)
- Mechanism and Emergence of Stacked Attention Heads in Multi-Layer Transformers (ICLR 2025 · similarity 0.88)
- Contrastive Retrieval Heads for Improved Attention-Based Reranking (NeurIPS 2026 · similarity 0.87)
- A Phase Transition between Positional and Semantic Learning in a Solvable Model of Dot-Product Attention (NeurIPS 2024 · similarity 0.87)
- MoH: Multi-Head Attention as Mixture-of-Head Attention (ICML 2025 · similarity 0.87)
- Bayes optimal learning of attention-indexed models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
