Research map / Architectures and efficiency / Attention and transformers
Vision-LSTM: xLSTM as Generic Vision Backbone
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Vision-LSTM (ViL), an adaptation of xLSTM/mLSTM blocks into a generic isotropic vision backbone that alternates traversal direction between blocks, evaluated on ImageNet-1K, ADE20K and VTAB-1K against ViT, Vim and other backbones.
Problem
Transformers are the default generic vision backbone but scale quadratically with sequence length, making high-resolution tasks costly; recent recurrent/SSM alternatives for vision need a way to handle non-causal 2D image inputs.
Approach
Adapt the xLSTM mLSTM block to vision by stacking residual mLSTM blocks in which the patch sequence is flipped in every second block (top-left to bottom-right alternating with bottom-right to top-left), and replace the causal 1D convolution with a 2D convolution, giving linear complexity in token count.
Key results
- ViL-T reaches 78.3% ImageNet-1K accuracy with 6M params and 1.3G FLOPS, above DeiT-III-T reimplementation (76.2%) and Vim-T (76.1%).
- ViL-S reaches 81.5% ImageNet-1K and 70.0% VTAB-1K average, outperforming DeiT-III-S (81.4%, 67.2%).
- ViL-B achieves 82.4% ImageNet-1K, below DeiT-III-B (83.7%), but 70.9% VTAB-1K average (vs 70.3%) and 48.6 mIoU on ADE20K single-scale (vs 47.5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use bidirectional instead of unidirectional traversal (ablation) | ViL-T · ImageNet-1K and ADE20K | accuracy / mIoU: 72.2 -> 73.7; 28.6 -> 31.7 | Helped |
| add column-wise traversal directions (quad-directional) (ablation) | ViL-T · ADE20K | mIoU: 31.7 -> 33.1 | Helped |
| increase traversal to oct-directional (ablation) | ViL-T · ImageNet-1K and ADE20K | accuracy / mIoU: 73.7 -> 73.5; 31.7 -> 32.4 | No clear effect |
| replace causal 1D QK convolution with Conv2D (ablation) | ViL-T · ImageNet-1K and ADE20K | accuracy / mIoU: 72.8 -> 73.7; 28.4 -> 31.7 | Helped |
| add positional embeddings (ablation) | ViL-T · ADE20K | mIoU: 31.0 -> 31.7 | Helped |
| apply both directions in parallel and halve the depth instead of sequentially (ablation) | ViL-T · ImageNet-1K and ADE20K | accuracy / mIoU: 73.7 -> 73.0; 31.7 -> 30.6 | Hurt |
| average first and last patches instead of concatenating them (ablation) | ViL-T · ImageNet-1K | accuracy: 73.7 -> 73.0 | Hurt |
| use all-patch average aggregation instead of bilateral concatenation (ablation) | ViL-T · ImageNet-1K and ADE20K | accuracy / mIoU: 73.7 -> 72.6; 31.7 -> 32.8 | Mixed |
| train for 800 instead of 400 epochs (ablation) | ViL-T · ImageNet-1K | accuracy: 77.2 -> 78.3 | Helped |
| train for 800 instead of 400 epochs (ablation) | ViL-T · VTAB-1K | accuracy: 67.8 -> 68.3 | Helped |
| train for 800 instead of 400 epochs (ablation) | ViL-T · ADE20K | mIoU: 40.9 -> 41.2 | Helped |
Surprising findings
- Averaging all patch tokens ([AVG]) improves segmentation but hurts classification, while concatenating first and last patch works best for classification.
- Quad-directional traversal gives the best segmentation results, but the implementation was not torch.compile compatible, so the final models use bi-directional traversal.
Limitations
- No optimized hardware (CUDA/Triton) kernel for mLSTM, so wall-clock runtime is slower than ViT and FLOPS are only a proxy.
- ViL-B does not outperform DeiT-III-B on ImageNet-1K classification.
- Hyperparameters and architecture received less tuning than ViT; quad/oct-directional traversal was not used in final models due to slow implementation.
Experimental setup
Datasets: ImageNet-1K ADE20K VTAB-1K
Benchmarks: ImageNet-1K ADE20K VTAB-1K ImageNet-C ImageNet-A ImageNet-R ImageNet-Sketch
Metrics: top-1 accuracy mIoU pixelwise accuracy FLOPS mean corruption error
Baselines: DeiT DeiT-III Vim VRWKV Mamba ConvNeXt
Builds on: xLSTM mLSTM Vision Transformer DeiT-III Vision Mamba
Closest papers by meaning
- xLSTM: Extended Long Short-Term Memory (NeurIPS 2024 · similarity 0.88)
- Sequencer: Deep LSTM for Image Classification (NeurIPS 2022 · similarity 0.88)
- LF-ViT: Reducing Spatial Redundancy in Vision Transformer for Efficient Image Recognition (AAAI 2024 · similarity 0.84)
- Linearly Decomposing and Recomposing Vision Transformers for Diverse-Scale Models (NeurIPS 2024 · similarity 0.84)
- RPFQ-ViT: Rotated Phase-Frame Quantization for Extremely Low-Bit Weights in Vision Transformers (NeurIPS 2026 · similarity 0.84)
- Sequential Modeling Enables Scalable Learning for Large Vision Models (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.