Research map / Architectures and efficiency / RNNs and normalization
A Constrained Optimization Perspective of Unrolled Transformers
Javier Porras-Valenzuela, Samar Hadou, Alejandro Ribeiro
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add layerwise descent constraints (primal-dual training) to RoBERTa/DistilBERT | RoBERTa, DistilBERT · IMDb with perturbed embeddings (OOD) | accuracy under OOD perturbation | Improved |
| add descent constraints to the unrolled transformer UT | UT · IMDb / MNLI with perturbed embeddings | accuracy | No clear effect |
| add descent constraints | Llama 3.1 8B SFT on Alpaca · AlpacaEval, gamma=0.0 | length-controlled win rate: 50% | No clear effect |
| add descent constraints | Llama 3.1 8B SFT on Alpaca · AlpacaEval, gamma=1.5 | length-controlled win rate: 69.93% win rate for constrained | Improved |
| increase constraint step size alpha (ablation) | constrained RoBERTa · IMDb | OOD accuracy: less than 0.3% in-distribution degradation | Improved |
| add descent constraints to ViT, DUST and UT | ViT, DUST, UT · Video denoising (Avenue, UCSD, ShanghaiTech, gamma_train=0.13) | OOD RMSE: better in five of nine cases, comparable in three, worse in one (UT-Avenue) | Mixed |
| increase number of layers L (ablation) | constrained DUST · UCSD video denoising | RMSE at high test gamma | Improved |
| increase number of layers L (ablation) | unconstrained DUST · UCSD video denoising | OOD RMSE | No clear effect |
| add descent constraints | UT · ShanghaiTech video denoising | ID RMSE vs OOD RMSE | Mixed |
Closest papers by meaning
- Unraveling the Gradient Descent Dynamics of Transformers (NeurIPS 2024 · similarity 0.86)
- Transformers Learn Low Sensitivity Functions: Investigations and Implications (ICLR 2025 · similarity 0.86)
- Linear Transformers are Versatile In-Context Learners (NeurIPS 2024 · similarity 0.86)
- Optimal Control for Transformer Architectures: Enhancing Generalization, Robustness and Efficiency (NeurIPS 2025 · similarity 0.86)
- Attention-Only Transformers via Unrolled Subspace Denoising (ICML 2025 · similarity 0.86)
- Unveil Benign Overfitting for Transformer in Vision: Training Dynamics, Convergence, and Generalization (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
