Research map / Architectures and efficiency / Attention and transformers
Understanding Differential Transformer Unchains Pretrained Self-Attentions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes why Differential Transformer works (negative attention, lower head redundancy, better optimization) and proposes Dex, a lightweight adaptation that retrofits differential-style output operations onto pretrained Llama/Qwen self-attention using under 1B tokens.
Problem
Differential Transformer's mechanisms are poorly understood, and its architecture requires training from scratch, which prevents reuse of open pretrained LLM weights.
Approach
Reuse pretrained softmax attention scores and apply a learnable differential operation on the output value product (O' = O - λ f_D(O)) to selected heads, with head selection by importance or entropy and a scheduled annealing of λ, updating only self-attention parameters.
Key results
- On 11 language modeling benchmarks, Dex on Llama-3.2-3B raises the average from 60.2 to 64.2 (+4.0), outperforming LoRA, PiSSA, FT, GaLore and APOLLO trained on the same corpus.
- On Needle-in-a-Haystack (N=8, R=1), Dex improves Llama-3B average accuracy from 66.9% to 78.3%.
- Dex adds negligible inference overhead, whereas Diff attention is substantially slower at long context.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Dex adaptation | Llama-3B · 11 language modeling benchmarks | average accuracy: 60.2 -> 64.2 | Helped |
| Apply Dex adaptation | Llama-1B · 11 language modeling benchmarks | average accuracy: 54.5 -> 55.6 | Helped |
| Apply Dex adaptation | Qwen-1.5B · 11 language modeling benchmarks | average accuracy: 62.2 -> 62.4 | Helped |
| Apply Dex adaptation | Qwen-0.5B · 11 language modeling benchmarks | average accuracy: 51.8 -> 54.4 | Helped |
| Apply Dex adaptation | Llama-8B · 11 language modeling benchmarks | average accuracy: 67.6 -> 69.6 | Helped |
| Apply Dex retrieval adaptation | Llama-3B · Multi-needle retrieval (N=8, R=1) | average accuracy: 66.9% -> 78.3% | Helped |
| Apply Dex adaptation | Llama · TREC | average accuracy across N-shot settings: 75.9 -> 78.9 | Helped |
| Apply Dex adaptation | Llama · Banking-77 | average accuracy across N-shot settings: 64.1 -> 67.0 | Helped |
| Apply Dex adaptation | Llama · Clinic-150 | average accuracy across N-shot settings: 65.6 -> 66.3 | Helped |
| Apply Dex in continued instruction tuning | Instruction-tuned Llama-3.1-8B · 8 instruction-tuning benchmarks | average score: 52.2 -> 55.6 | Helped |
| Apply Dex as direct instruction tuning | Llama-3.1-8B · 8 instruction-tuning benchmarks | average score: 47.8 -> 53.9 | Helped |
| Select high-entropy heads instead of adapting all heads (ablation) | Dex on Llama-3B · 11 language modeling benchmarks | average accuracy: 61.9 -> 64.2 | Helped |
Surprising findings
- Diff Transformer's attention magnitudes correlate strongly with softmax attention, and its advantage is attributed to negative attention and non-sparse patterns rather than traditional sparsity.
- Merely using fewer, wider heads does not reduce inter-head redundancy, so head count alone does not explain Diff's benefit.
Limitations
- Not tested beyond 8B parameter scale.
- Diff Transformer analysis uses a self-trained 0.4B model because original weights were unavailable; behaviors may differ at other scales or data mixes.
- Some benchmarks degrade after adaptation given the mismatch between training corpus and downstream tasks.
Experimental setup
Datasets: Dolmino-mix subset (web pages, papers, encyclopedia, code) OpenHermes-2.5
Benchmarks: Arc-C Arc-E BoolQ COPA Hellaswag MNLI OBQA PIQA WIC Winogrande WSC Needle-in-a-Haystack
Metrics: accuracy attention to answer ratio throughput latency
Baselines: LoRA PiSSA FT GaLore APOLLO Diff Transformer DIFFT-Lam
Builds on: Differential Transformer MHA2MLA LoRA PiSSA GaLore APOLLO
Closest papers by meaning
- Differential Transformer (ICLR 2025 · similarity 0.90)
- The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training (ICML 2025 · similarity 0.87)
- Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers (Student Abstract) (AAAI 2024 · similarity 0.87)
- Neural ODE Transformers: Analyzing Internal Dynamics and Adaptive Fine-tuning (ICLR 2025 · similarity 0.87)
- The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry (ICLR 2024 · similarity 0.87)
- Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscape (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.