Research map / Architectures and efficiency / Attention and transformers

Theory, Analysis, and Best Practices for Sigmoid Self-Attention

ICLR 2025 · Architectures and efficiency › Attention and transformers · approach: Attention head and block redesign · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Revisits sigmoid attention and derives a variance-preservation argument for choosing the output scaling power (alpha = 1/2 under data-independent weights), with best practices for making sigmoid attention work in transformers.

Problem

Sigmoid attention replaces softmax in transformers, but its output scaling (how the sum over n tokens should be normalized, n^alpha) is unclear, and naive use can cause n-dependent growth of output variance.

Approach

Uses a variance-propagation analysis of sigmoid attention outputs: with data-independent attention weights the output variance scales as n^(1-2alpha), so alpha=1/2 preserves variance; with data-dependent weights the analysis is more involved, and the paper extends this into best practices for sigmoid attention.

Key results

• Under data-independent attention weights, Var(y_i) scales as n^(1-2alpha) times Var(a)Var(x), so alpha=1/2 (sqrt scaling) cancels n-dependent growth.
• With data-dependent attention weights, the variance expression retains covariance cross-terms between attention and inputs, so the simple sqrt-scaling result does not directly apply.

Limitations

• Variance analysis is simplified to d=1 and W_v=1 with zero-centered random variables.
• The data-dependent case is not resolved to a closed-form choice of alpha in the shown text.

Experimental setup

Baselines: softmax attention

Builds on: softmax attention transformer

Closest papers by meaning

• Transformers Learn Faster with Semantic Focus (NeurIPS 2025 · similarity 0.90)
• ZeroS: Zero‑Sum Linear Attention for Efficient Transformers (NeurIPS 2025 · similarity 0.89)
• LASER: Attention with Exponential Transformation (ICML 2025 · similarity 0.89)
• Memorization Capacity of Multi-Head Attention in Transformers (ICLR 2024 · similarity 0.89)
• The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry (ICLR 2024 · similarity 0.89)
• Mixture-of-Top-$k$ Attention: Efficient Attention as Scalable Fast Weights (NeurIPS 2026 · similarity 0.89)