Research map / Multi-agent systems and games / Human-AI and goal-conditioned

A Theoretical Study of (Hyper) Self-Attention through the Lens of Interactions: Representation, Training, Generalization

ICML 2025 · Multi-agent systems and games › Human-AI and goal-conditioned · approach: Language-grounded reinforcement agents · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Studies single-layer linear self-attention as a mutual-interaction learner, proving exact representation, gradient-flow convergence to zero training error, and in-distribution and length generalization, then proposes HyperFeatureAttention and HyperAttention with small language-modeling experiments.

Problem

Theoretical understanding of what self-attention can represent, learn via gradient descent, and generalize (including to other sequence lengths), especially for functions capturing pairwise and higher-order interactions among entities.

Approach

Views tokens as interacting entities and shows a single linear self-attention layer with orthonormal embeddings (d = domain size) exactly represents pairwise interaction functions; gradient flow converges to zero training error under a training-data versatility (full column rank) assumption, and the learned parameters generalize to other sequence lengths. Extends attention with Hadamard-product co

Key results

• On the colliding agents task, linear self-attention achieves test error on the order of 1e-7 both at L=20 and when L varies over {2,5,10,30,40}, consistent with the theory.
• In a 1-layer, 1-head benchmark, perplexity drops from 62.28 (self-attention) to 60.22 (HyperFeatureAttention, order 4) and 51.26 (HyperAttention, order 3), and 48.50 for HyperAttention without parameter sharing.
• In a 3-layer, 1024-token benchmark, a hybrid HFA variant reaches perplexity 27.75 versus 28.70 for self-attention.

Measured effects reported in the paper

Surprising findings

• Learned parameters in the sinusoidal setting do not match the originally devised parameters directly, but match after a corollary-based transformation that yields functionally equivalent length-generalizing solutions.

Limitations

• Theory assumes embedding dimension equals domain size, which is impractical for large vocabularies.
• Analysis restricted to linear self-attention without softmax and with noise-free realizable data.
• Language-modeling experiments are small-scale and preliminary.

Experimental setup

Datasets: OpenWebText colliding agents synthetic data

Benchmarks: OpenWebText validation perplexity colliding agents value function

Metrics: mean squared error perplexity

Baselines: Self-Attention HyperAttention (Zhang et al.-style low-rank higher-order attention)

Builds on: linear self-attention theory GPT-3 small training setup Performer

Closest papers by meaning

• Attention as a Hypernetwork (ICLR 2025 · similarity 0.86)
• The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training (ICML 2025 · similarity 0.86)
• Linking In-context Learning in Transformers to Human Episodic Memory (NeurIPS 2024 · similarity 0.85)
• Training Dynamics of In-Context Learning in Linear Attention (ICML 2025 · similarity 0.85)
• Hierarchical Self-Attention: Generalizing Neural Attention Mechanics to Multi-Scale Problems (NeurIPS 2025 · similarity 0.85)
• Inverse Attention Agents for Multi-Agent Systems (ICLR 2025 · similarity 0.85)