Research map / Architectures and efficiency / Attention and transformers
Attention as a Hypernetwork
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reformulates multi-head attention as a hypernetwork whose head-indexed attention scores form a latent code, and proposes HYLA (nonlinear value network with RMSHead normalization) that improves compositional generalization on fuzzy logic and the new symbolic Raven task (SRAVEN).
Problem
How transformers achieve compositional generalization in-context to unseen combinations of known constituents, and what internal mechanism supports it.
Approach
Multi-head attention is mathematically equivalent to a linear hypernetwork: per key-query pair, the head-indexed attention scores act as a latent code that linearly combines shared head matrices into a key-query-specific value network. Making this value network nonlinear (HYLA, with RMSHead normalization across heads) without extra parameters strengthens compositional generalization.
Key results
- On SRAVEN with 4-layer transformer trained on 20M instances, HYLA reaches 69.13% OOD accuracy vs 56.56% for softmax attention and 56.30% for linear attention.
- On the fuzzy logic task (70% held out, length 32), HYLA achieves 81.13 OOD R^2 vs 63.28 for softmax and 59.89 for linear attention; deeper HYLA+ reaches 87.65.
- Latent codes of the final layer cluster by SRAVEN rule and are predictive of the ground-truth rule on unseen OOD tasks; HYLA improves over linear attention on C4 language modeling and performs close to softmax.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HYLA instead of softmax or linear attention | Transformer attention · Fuzzy logic OOD tasks | OOD R2: 63.28 ± 2.31 -> 81.13 ± 7.77; 59.89 ± 5.22 -> 81.13 ± 7.77 | Helped |
| Use HYLA instead of softmax or linear attention | Transformer attention · SRaven OOD tasks | OOD accuracy: 56.56 ± 1.05 -> 69.13 ± 1.90; 56.30 ± 1.11 -> 69.13 ± 1.90 | Helped |
| Add RMSHead normalization to linear attention (ablation) | Linear attention · Fuzzy logic OOD tasks | OOD R2: 59.89 ± 5.22 -> 68.93 ± 5.10 | Helped |
| Add RMSHead normalization to linear attention (ablation) | Linear attention · SRaven OOD tasks | OOD accuracy: 56.30 ± 1.11 -> 59.01 ± 2.50 | Helped |
| Add a nonlinear value network to Linear attention + RMSHead (ablation) | Linear attention · Fuzzy logic OOD tasks | OOD R2: 68.93 ± 5.10 -> 56.91 ± 4.25 | Hurt |
| Add a nonlinear value network to Linear attention + RMSHead (ablation) | Linear attention · SRaven OOD tasks | OOD accuracy: 59.01 ± 2.50 -> 55.38 ± 0.32 | Hurt |
| Add RMSHead normalization (ablation) | HYLA · SRaven OOD tasks | OOD accuracy: 62.53 ± 5.73 -> 69.13 ± 1.90 | Helped |
| Add the value-network nonlinearity (ablation) | HYLA · Fuzzy logic OOD tasks | OOD R2: 81.13 ± 7.77 -> 84.54 ± 5.32 | Hurt |
| Add the value-network nonlinearity (ablation) | HYLA · SRaven OOD tasks | OOD accuracy: 56.54 ± 0.21 -> 69.13 ± 1.90 | Helped |
| Replace RMSHead normalization with softmax (ablation) | HYLA · SRaven OOD tasks | OOD accuracy: 69.13 ± 1.90 -> 38.21 ± 2.98 | Hurt |
| Add an additional value-network layer (ablation) | HYLA · Fuzzy logic OOD tasks | OOD R2: 81.13 ± 7.77 -> 87.65 ± 4.54 | Helped |
| Add an additional value-network layer (ablation) | HYLA · SRaven OOD tasks | OOD accuracy: 69.13 ± 1.90 -> 66.59 ± 0.08 | Hurt |
Surprising findings
- Fuzzy logic models fail to generalize to functions made of novel combinations of unseen terms, none of the models solve this.
- Ablating the nonlinearity alone (HYLA minus nonlinearity) achieves higher fuzzy logic OOD R^2 (84.54) than full HYLA (81.13), though it is worse on SRAVEN.
Limitations
- Analysis focuses on models trained from scratch, not large pretrained models.
- Compositional generalization fails for novel combinations of unseen terms.
- Symbolic SRAVEN removes visual representation learning and is not directly compared to image-based Raven benchmarks.
Experimental setup
Datasets: SRAVEN (generated) fuzzy logic functions (generated) C4
Benchmarks: SRAVEN fuzzy logic compositional split C4 language modeling
Metrics: OOD accuracy OOD R^2 perplexity / language modeling loss F1 of logistic regression probes on latent codes
Baselines: softmax attention linear attention Linear attention + RMSHead HYLA deep value network Mixture of experts
Builds on: hypernetworks linear attention as fast weight programmer Raven's Progressive Matrices T5 relative positional embeddings
Closest papers by meaning
- Scaling can lead to compositional generalization (NeurIPS 2025 · similarity 0.90)
- CraftFactory: A Conditioned Control Policy Benchmark for Compositional Generalization (AAAI 2025 · similarity 0.88)
- Compositional Generalization from Learned Skills via CoT Training: A Theoretical and Structural Analysis for Reasoning (ICLR 2026 · similarity 0.87)
- Are Transformers Able to Reason by Connecting Separated Knowledge in Training Data? (ICLR 2025 · similarity 0.87)
- Separating the 'what' and 'how' of compositional computation to enable reuse and continual learning (NeurIPS 2025 · similarity 0.87)
- Mechanism and Emergence of Stacked Attention Heads in Multi-Layer Transformers (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.