Research map / Architectures and efficiency / Attention and transformers

ZeroS: Zero‑Sum Linear Attention for Efficient Transformers

NeurIPS 2025 · Architectures and efficiency › Attention and transformers · approach: Linear attention approximations · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ZeroS, a linear-time attention that removes the constant 1/t zero-order softmax term and reweights the zero-sum residuals with learned gates, enabling signed weights and contrastive operations.

Problem

Linear attention is O(N) but typically underperforms softmax attention; convex-combination weights only permit additive blending and accumulate uniform bias that dilutes attention in long contexts.

Approach

Subtract the uniform 1/t zero-order softmax term and the first-order deviation, then gate the resulting zero-sum residuals with learned sigmoid gates and multiply by a signed cosine angular term. Prefix-sum scans of three d x d states make this linear time while allowing negative weights.

Key results

• On MAD, ZeroS reaches 73.5 average versus 74.5 for softmax Transformer, and ZeroS-SM reaches 75.4.
• On WikiText-103, ZeroS obtains 24.61 test perplexity versus 24.78 for the Transformer, and ZeroS-SM obtains 24.17.
• On ImageNet-1k with DeiT-Tiny, ZeroS reaches 75.51% top-1 versus 72.20% for DeiT.

Measured effects reported in the paper

Surprising findings

• Reintroducing the 0-th order softmax term degraded performance on MAD and WikiText-103, contrary to the expectation that the constant term aids stability.

Limitations

• No fused CUDA kernel; implementation invokes an existing linear-attention scan three times.
• No large-scale LLM training or LLM benchmark evaluation due to resource constraints.
• Evaluation focuses primarily on autoregressive tasks; non-causal tasks left for future work.

Experimental setup

Datasets: MAD MQAR RegBench WikiText-103 OpenWebText2 ImageNet-1k Weather Solar ETT

Benchmarks: MAD MQAR RegBench WikiText-103 OpenWebText2 ImageNet-1k time series forecasting benchmarks

Metrics: accuracy perplexity top-1 accuracy MSE MAE latency throughput memory

Baselines: Hyena Mamba GLA DeltaNet LinAttn Transformer FLASH cosFormer HGRN2 GatedDeltaNet Mamba2 iTransformer

Builds on: linear attention RoPE Taylor expansion of softmax Differential Transformer

Closest papers by meaning

• Theory, Analysis, and Best Practices for Sigmoid Self-Attention (ICLR 2025 · similarity 0.89)
• Degrees of Freedom for Linear Attention: Distilling Softmax Attention with Optimal Feature Efficiency (NeurIPS 2025 · similarity 0.89)
• Bridging the Divide: Reconsidering Softmax and Linear Attention (NeurIPS 2024 · similarity 0.89)
• Mixture-of-Top-$k$ Attention: Efficient Attention as Scalable Fast Weights (NeurIPS 2026 · similarity 0.88)
• MonarchAttention: Zero-Shot Conversion to Fast, Hardware-Aware Structured Attention (NeurIPS 2025 · similarity 0.87)
• Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction (ICLR 2025 · similarity 0.87)