Research map / Architectures and efficiency / Attention and transformers

Dynamic Convolutions Improve Transformers

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Transformer block and depth design

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes dynamic short convolutions (input-dependent depthwise filters) applied to Transformer Q/K/V or all linear layers, with Triton kernels, showing gains on MQAR, language modeling from 150M to 2B dense and a 7B MoE, and linear RNNs.

Problem

Static short convolutions add locality but cannot adapt filters to context, limiting expressivity for local context-dependent composition; it is unclear whether a scalable, hardware-efficient input-dependent convolution primitive improves Transformer LLMs.

Approach

Generate the depthwise short-convolution filter at each position from the current hidden state (low-rank or head-wise parameterization), applied with residuals to Q, K, V or all linear layers, and implement fused Triton kernels to make it efficient.

Key results

• Scaling laws give 1.33x compute advantage for low-rank dynamic conv on QKV and 1.60x for all-linear placement over compute-matched Transformers.
• On the 2B dense model at 100B tokens, all-linear dynamic conv reaches 60.70 task avg vs 58.35 for Transformer.
• End-to-end throughput slowdown is under 8% for QKV variants and 22-25% for the all-linear variant on a single H100.

Surprising findings

• Static short convolutions give little benefit when combined with QK-norm, while dynamic convolutions still give significant gains.
• Mamba-2 with dynamic convolutions performs about as well as Gated DeltaNet with static convolutions in perplexity.

Limitations

• Scaling study limited to 2B dense and a 7B MoE (1B active); unclear if advantage holds at frontier scale or longer training.
• All-linear placement incurs 22-25% training throughput overhead; inference not optimized.
• Only a small subset of parameterizations, placements, and kernel widths explored.

Experimental setup

Datasets: Nemotron-CC Wikitext-103 LAMBADA RULER MAD

Benchmarks: Wikitext-103 LAMBADA RULER MAD lm-eval-harness common-sense tasks MQAR

Metrics: perplexity zero-shot accuracy accuracy compute advantage from scaling laws training throughput (tokens/s)

Baselines: Transformer Transformer with static short convolutions Transformer (more params) Gated DeltaNet Mamba-2 MoE Transformer

Builds on: Mamba-2 Gated DeltaNet MQAR Canon layers lm-engine

Closest papers by meaning

• On the Power of Convolution-Augmented Transformer (AAAI 2025 · similarity 0.90)
• Viewing Transformers Through the Lens of Long Convolutions Layers (ICML 2024 · similarity 0.87)
• LeaPformer: Enabling Linear Transformers for Autoregressive and Simultaneous Tasks via Learned Proportions (ICML 2024 · similarity 0.87)
• Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory (AAAI 2024 · similarity 0.87)
• Frozen Transformers in Language Models Are Effective Visual Encoder Layers (ICLR 2024 · similarity 0.86)
• KernelWarehouse: Rethinking the Design of Dynamic Convolution (ICML 2024 · similarity 0.86)