Research map / Architectures and efficiency / Attention and transformers
Disentangling and Integrating Relational and Sensory Information in Transformer Architectures
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the Dual Attention Transformer (DAT), which adds relational attention heads alongside standard sensory attention heads so that relational information is routed and processed explicitly, and reports gains on relational games, math problem solving, CIFAR image classification, and language modeling.
Problem
Transformers struggle with relational reasoning because standard attention entangles relational information with sensory features and has no explicit mechanism for routing and processing relations between objects.
Approach
Relational attention routes relation vectors (inner-product comparisons under learned feature maps) tagged with abstract symbols identifying the source, instead of sensory values; it is combined with standard attention heads in a multi-head "dual attention" block that is used inside Transformer encoder/decoder blocks.
Key results
- On CIFAR-10 the ViDAT model (6.0M params) reaches 89.7% vs 86.4% for ViT (7.1M params); on CIFAR-100 70.5% vs 68.8%.
- DAT is more sample-efficient than parameter-matched Transformers on Relational Games, with the largest gap on the match pattern task.
- DAT outperforms a standard Transformer at all tested model scales on mathematical problem-solving tasks, and scales to 1.3B parameters on FineWeb-Edu with better data and parameter efficiency.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add relational attention heads and use symmetric relations | ViT · CIFAR-10 | accuracy: 86.4 ± 0.1% -> 89.7 ± 0.1% | Helped |
| Add relational attention heads and use symmetric relations | ViT · CIFAR-100 | accuracy: 68.8 ± 0.2% -> 70.5 ± 0.1% | Helped |
| Use relational attention instead of standard sensory attention | Transformer · Relational Games | sample efficiency | Helped |
| Use relational-only attention heads instead of a mixture of sensory and relational heads (ablation) | DAT · Relational Games | performance | Helped |
| Add relational attention to the Transformer | Transformer encoder-decoder · Mathematical problem-solving tasks | character-level accuracy | Helped |
| Add relational attention to the Transformer | Transformer decoder-only language model · FineWeb-Edu language modeling | language-modeling performance | Helped |
| Add relational attention to the Transformer | Transformer · Relational Games match pattern task | sample efficiency | Helped |
Surprising findings
- Relational-only DAT heads perform slightly better than the mixed sensory-plus-relational variant on the purely relational match pattern task.
Limitations
- Implementation lacks hardware-aware optimizations (e.g., FlashAttention), making training slower.
- Introduces several hyperparameters and configuration choices (head composition, symmetry, symbol assignment) that were only partly ablated.
- Relational benefits on complex real-world tasks such as scene parsing are not evaluated.
Experimental setup
Datasets: Relational Games mathematical problem-solving benchmark (Saxton et al.) CIFAR-10 CIFAR-100 FineWeb-Edu
Benchmarks: Relational Games mathematical problem-solving tasks CIFAR-10 CIFAR-100 FineWeb-Edu language modeling
Metrics: sample efficiency learning-curve accuracy character-level accuracy classification accuracy language modeling loss
Baselines: Transformer ViT PrediNet CoRelNet Abstractor
Builds on: Transformer Vision Transformer PrediNet CoRelNet Abstractor
Closest papers by meaning
- Beyond the Doors of Perception: Vision Transformers Represent Relations Between Objects (NeurIPS 2024 · similarity 0.88)
- What are you sinking? A geometric approach on attention sink (NeurIPS 2025 · similarity 0.88)
- Beyond Components: Singular Vector-Based Interpretability of Transformer Circuits (NeurIPS 2025 · similarity 0.87)
- The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training (ICML 2025 · similarity 0.87)
- Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers (Student Abstract) (AAAI 2024 · similarity 0.87)
- Associative Transformer (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.