Research map / Architectures and efficiency / Attention and transformers
LipsFormer: Introducing Lipschitz Continuity to Vision Transformers
Xianbiao Qi, Jianan Wang, Yihao Chen, Yukai Shi, Lei Zhang
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the CSwin-T design with LipsFormer components and train without warmup | CSwin-T · ImageNet-1K | Top-1 accuracy: 82.7 -> 83.5 | Improved |
| Replace Swin-T with LipsFormer-Swin-T | Swin-T · ImageNet-1K | accuracy: 81.3 -> 82.7 | Improved |
| Add 5 epochs of learning-rate warmup (ablation) | LipsFormer-CSwin-T · ImageNet-1K, 100 epochs | Top-1 accuracy: 81.6 -> 81.2 | Hurt |
| Add CenterNorm instead of using no normalization (ablation) | LipsFormer-CSwin-T · ImageNet-1K, 100 epochs | Top-1 accuracy: 80.3 -> 81.6 | Improved |
| Replace CenterNorm with LayerNorm (ablation) | LipsFormer-CSwin-T · ImageNet-1K, 100 epochs | convergence | Hurt |
| Use spectral initialization instead of truncated-normal initialization (ablation) | LipsFormer-CSwin-T · ImageNet-1K, 100 epochs | Top-1 accuracy: 81.3 -> 81.6 | Improved |
| Replace scaled cosine similarity attention with dot-product attention (ablation) | LipsFormer-CSwin-T · ImageNet-1K, 100 epochs | convergence | Hurt |
| Use scaled cosine similarity attention instead of L2 distance attention (ablation) | LipsFormer-CSwin-T · ImageNet-1K, 100 epochs | Top-1 accuracy: 81.3 -> 81.6 | Improved |
| Increase residual weight alpha from 0.1 to 0.2 (ablation) | LipsFormer-CSwin-T · ImageNet-1K, 100 epochs | Top-1 accuracy: 81.4 -> 81.6 | Improved |
| Increase residual weight alpha from 0.2 to 0.4 (ablation) | LipsFormer-CSwin-T · ImageNet-1K, 100 epochs | convergence | Hurt |
| Add convolution blocks, from none to one depth-wise convolution (ablation) | LipsFormer-CSwin-T · ImageNet-1K, 100 epochs | Top-1 accuracy: 80.7 -> 81.2 | Improved |
| Add a point-wise convolution after one depth-wise convolution (ablation) | LipsFormer-CSwin-T · ImageNet-1K, 100 epochs | Top-1 accuracy: 81.2 -> 81.6 | Improved |
Closest papers by meaning
- LipNeXt: Scaling up Lipschitz-based Certified Robustness to Billion-parameter Models (ICLR 2026 · similarity 0.88)
- Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformers (ECCV 2026 · similarity 0.88)
- Beyond Uniformity: Regularizing Implicit Neural Representations through a Lipschitz Lens (ICLR 2026 · similarity 0.87)
- Efficient Bound of Lipschitz Constant for Convolutional Layers by Gram Iteration (ICML 2023 · similarity 0.87)
- Some Fundamental Aspects about Lipschitz Continuity of Neural Networks (ICLR 2024 · similarity 0.86)
- ECLipsE: Efficient Compositional Lipschitz Constant Estimation for Deep Neural Networks (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
