Research map / Architectures and efficiency / Attention and transformers
Differential Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Diff Transformer, which computes attention as the difference of two softmax attention maps to cancel attention noise, and shows gains in scaling, long-context retrieval, hallucination, in-context learning, and activation outliers.
Problem
Transformers overallocate attention to irrelevant context ("attention noise"), which hurts retrieval of key information, contextual hallucination, in-context learning robustness, and produces large activation outliers.
Approach
Differential attention takes the difference between two softmax attention maps (with a learnable, reparameterized scalar lambda), analogous to differential amplifiers and noise-canceling headphones, so common-mode attention noise cancels and sparse attention patterns emerge; it can reuse FlashAttention.
Key results
- Diff 6.8B matches an 11B Transformer's validation loss using 62.2% of parameters; Diff trained on 160B tokens matches Transformer at 251B tokens (63.7%).
- Multi-needle retrieval at 4K (N=6, R=2): Diff 0.85 vs Transformer 0.55, a 30-point accuracy gap.
- Hallucination accuracy on XSum/MultiNews/Qasper improves (e.g., MultiNews 0.42 to 0.61; Qasper 0.28 to 0.39), and 4-bit Diff roughly matches 6-bit Transformer on HellaSwag.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace softmax attention with differential attention | 3B Transformer · LM Eval Harness, 1T tokens | average accuracy: 57.5 -> 60.6 | Helped |
| Replace softmax attention with differential attention | 3B Transformer · Multi-needle retrieval, 4K context, N=6, R=2 | accuracy: 0.55 -> 0.85 | Helped |
| Replace softmax attention with differential attention | 3B Transformer · Contextual hallucination, MultiNews | accuracy free of hallucinations: 0.42 -> 0.61 | Helped |
| Use 6.8B Diff Transformer instead of 11B Transformer | language modeling scaling · Validation loss | validation loss: 62.2% of parameters | Helped |
| Use 160B training tokens with Diff Transformer instead of 251B with Transformer | 3B language model · Language modeling scaling | language modeling loss: 63.7% of the training tokens | Helped |
| Replace softmax attention with differential attention | 3B Transformer · Contextual hallucination, summarization and question answering | accuracy free of hallucinations | Helped |
| Add GroupNorm to differential attention (ablation) | Diff Transformer · 1.4B language model validation set | validation loss: 3.122 -> 3.062 | Helped |
| Add GroupNorm to Transformer (ablation) | Transformer, 8 heads and d=256 · 1.4B language model validation set | validation loss: 3.088 -> 3.086 | No clear effect |
| Use 8 heads instead of 16 heads (ablation) | Transformer · 1.4B language model validation set | validation loss: 3.087 -> 3.088 | No clear effect |
| Set lambda initialization to 0.8 instead of the layer-dependent default (ablation) | Diff Transformer · 1.4B language model validation set | validation loss: 3.062 -> 3.065 | No clear effect |
| Set lambda initialization to 0.5 instead of the layer-dependent default (ablation) | Diff Transformer · 1.4B language model validation set | validation loss: 3.062 -> 3.066 | No clear effect |
Surprising findings
- Differential attention reduces top attention-logit outliers drastically (Transformer top-1 318.0 vs Diff 38.8), enabling more robust low-bit quantization.
- Throughput cost is modest, about 6-12% lower for 3B models versus Transformer.
Limitations
- Throughput is somewhat lower than standard Transformer (about 5-12%) without specialized kernels.
- Long-context and math results are limited to 3B models and selected baselines.
- Hallucination evaluation relies on GPT-4o binary judgments on 100 samples per dataset.
Experimental setup
Datasets: StableLM-3B-4E1T training corpus book data OpenThoughts-114K-Math synthetic math data LongBench XSum CNN/DM MultiNews Qasper HotpotQA 2WikiMultihopQA TREC
Benchmarks: LM Eval Harness (ARC-C, ARC-E, BoolQ, HellaSwag, OBQA, PIQA, WinoGrande) Needle-In-A-Haystack multi-needle TREC TREC-fine Banking-77 Clinic-150 XSum CNN/DM MultiNews Qasper HotpotQA 2WikiMultihopQA
Metrics: accuracy validation loss negative log-likelihood attention score activation magnitude throughput (tokens/s) variance across demonstration orderings
Baselines: Transformer OpenLLaMA-v2-3B StableLM-base-alpha-3B-v2 StableLM-3B-4E1T
Builds on: Transformer LLaMA FlashAttention RMSNorm SwiGLU
Closest papers by meaning
- Understanding Differential Transformer Unchains Pretrained Self-Attentions (NeurIPS 2025 · similarity 0.90)
- Selective Attention Improves Transformer (ICLR 2025 · similarity 0.87)
- From Attention to Activation: Unraveling the Enigmas of Large Language Models (ICLR 2025 · similarity 0.87)
- The Closeness of In-Context Learning and Weight Shifting for Softmax Regression (NeurIPS 2024 · similarity 0.87)
- Selective Attention: Enhancing Transformer through Principled Context Control (NeurIPS 2024 · similarity 0.86)
- Transformers Get Stable: An End-to-End Signal Propagation Theory for Language Models (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.