Research map / Architectures and efficiency / Attention and transformers
PIDformer: Transformer Meets Control Theory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PIDformer, a transformer whose attention is a discretized PID-controlled state-space model that adds proportional, integral, and derivative feedback on a reference signal, improving robustness and mitigating rank collapse.
Problem
Softmax transformers suffer from sensitivity to input corruption and rank collapse in deep layers, where token representations become increasingly similar.
Approach
Interprets self-attention as an autonomous state-space model (gradient flow minimizing nonlocal total variation) and adds a PID feedback controller with a reference point scaled from the initial value embedding, yielding a PID-attention that keeps the steady state sensitive only to a scaled input and preserves rank.
Key results
- PID DeiT reaches 73.13% clean top-1 on ImageNet vs 72.17% for softmax DeiT, and 38.52% vs 33.64% under FGSM.
- On ADE20K, PID DeiT-backed Segmenter improves SS MIoU from 35.72 to 37.42 and MS MIoU from 36.68 to 38.28.
- PIDformer language model lowers WikiText-103 test perplexity from 34.29 to 33.45.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add PID control to DeiT-tiny | DeiT-tiny · ImageNet validation (clean) | Top-1 accuracy (%): 72.17 -> 73.13 | Helped |
| Add PID control to DeiT-tiny | DeiT-tiny · ImageNet validation (FGSM) | Top-1 accuracy (%): 33.64 -> 38.52 | Helped |
| Add PID control to DeiT-tiny | DeiT-tiny · ImageNet validation (PGD) | Top-1 accuracy (%): 12.02 -> 15.08 | Helped |
| Add PID control to DeiT-tiny | DeiT-tiny · ImageNet validation (SPSA) | Top-1 accuracy (%): 65.75 -> 67.98 | Helped |
| Add PID control to DeiT-tiny | DeiT-tiny · ImageNet validation (SLD) | Top-1 accuracy (%): 69.32 -> 70.84 | Helped |
| Add PID control to DeiT-tiny | DeiT-tiny · ImageNet validation (noise) | Top-1 accuracy (%): 69.2 -> 70.87 | Helped |
| Add PID control to DeiT-tiny | DeiT-tiny · ImageNet-A | Top-1 accuracy (%): 6.90 -> 8.82 | Helped |
| Add PID control to DeiT-tiny | DeiT-tiny · ImageNet-R | Top-1 accuracy (%): 32.83 -> 34.89 | Helped |
| Add PID control to DeiT-tiny | DeiT-tiny · ImageNet-C | mCE (↓): 71.20 -> 68.41 | Helped |
| Add PID control to DeiT-tiny | DeiT-tiny · ImageNet-O | AUPR: 17.47 -> 19.22 | Helped |
| Add PID control to the DeiT-tiny backbone | Segmenter with DeiT-tiny backbone · ADE20K | SS mIoU: 35.72 -> 37.42 | Helped |
| Add PID control to the DeiT-tiny backbone | Segmenter with DeiT-tiny backbone · ADE20K | MS mIoU: 36.68 -> 38.28 | Helped |
Limitations
- Privacy-preserving aspects of PIDformer are not studied, as acknowledged by the authors.
- Theoretical analysis assumes a time-invariant attention kernel K, which is an approximation for transformers.
- Experiments are limited to DeiT-tiny-scale models and single-run comparisons with small gains.
Experimental setup
Datasets: ImageNet-1K ADE20K WikiText-103
Benchmarks: ImageNet-1K ImageNet-C ImageNet-A ImageNet-R ImageNet-O ADE20K WikiText-103
Metrics: Top-1 accuracy Top-5 accuracy mCE AUPR MIoU perplexity average cosine similarity
Baselines: Softmax DeiT Softmax Transformer Segmenter with DeiT backbone
Builds on: DeiT Segmenter Transformer-XL-style lmtool (fast weight programmers) Bregman iteration
Closest papers by meaning
- Optimal Control for Transformer Architectures: Enhancing Generalization, Robustness and Efficiency (NeurIPS 2025 · similarity 0.87)
- Unraveling the Gradient Descent Dynamics of Transformers (NeurIPS 2024 · similarity 0.87)
- Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding (NeurIPS 2025 · similarity 0.84)
- On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.84)
- Unveil Benign Overfitting for Transformer in Vision: Training Dynamics, Convergence, and Generalization (NeurIPS 2024 · similarity 0.84)
- MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connections (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.