Research map / Architectures and efficiency / Attention and transformers
Long-Short Transformer: Efficient Transformers for Language and Vision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine dynamic projection with local window attention | Dynamic Projection · ListOps | accuracy (%): 37.79 -> 38.36 | Helped |
| Combine dynamic projection with local window attention | Dynamic Projection · Text | accuracy (%): 66.28 -> 68.40 | Helped |
| Combine dynamic projection with local window attention | Dynamic Projection · Retrieval | accuracy (%): 81.86 -> 81.95 | No clear effect |
| Use dynamic projection instead of Linformer’s fixed projection | Linformer · Text | accuracy (%): 56.12 -> 66.28 | Helped |
| Use dynamic projection with local window attention | RoBERTa-large · IMDb | accuracy (%): 96.5 -> 96.8 | Helped |
| Use dynamic projection with local window attention | Longformer · enwik8 test | BPC: 1.00 -> 0.99 | Helped |
| Replace ViL attention with long-short attention | ViL-Base · ImageNet-1K 224² | top-1 accuracy (%): 83.7 -> 84.1 | Helped |
| Replace CvT attention with long-short attention | CvT-21 · ImageNet-1K 224² | top-1 accuracy (%): 82.5 -> 82.7 | Helped |
| Train ViL-LS-Medium at 384² instead of 224² | ViL-LS-Medium · ImageNet-1K | top-1 accuracy (%): 83.8 -> 84.4 | Helped |
| Add DualLN to align the local and global attention scales (ablation) | CvT*-LS-13 · ImageNet-1K validation | top-1 accuracy (%): 81.3 -> 81.9 | Helped |
| Use a longer warmup schedule | LRA models · Retrieval | accuracy | Helped |
| Increase model depth and width in early high-resolution stages | CvT*-LS-21 · ImageNet-1K 224² | top-1 accuracy (%): 82.7 -> 82.9 | Helped |
Closest papers by meaning
- Frozen Transformers in Language Models Are Effective Visual Encoder Layers (ICLR 2024 · similarity 0.90)
- Viewing Transformers Through the Lens of Long Convolutions Layers (ICML 2024 · similarity 0.90)
- LASER: Attention with Exponential Transformation (ICML 2025 · similarity 0.89)
- Dynamic Convolutions Improve Transformers (NeurIPS 2026 · similarity 0.88)
- SeTformer Is What You Need for Vision and Language (AAAI 2024 · similarity 0.88)
- LaVin-DiT: Large Vision Diffusion Transformer (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.