Research map / Architectures and efficiency / Attention and transformers
Spectral State Space Model for Rotation-Invariant Visual Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Spectral VMamba, which orders image patches for a visual state space model using spectral decomposition of a patch graph Laplacian plus a Rotational Feature Normalizer, giving rotation-invariant traversal and better classification than VMamba on miniImageNet.
Problem
Vision state space models (e.g., VMamba) scan patches along fixed horizontal/vertical routes, so they miss relations between non-adjacent but related patches and their patch ordering changes under image rotation.
Approach
Replace fixed scan routes with a Spectral Traversal Scan that orders patches by eigenvectors of the normalized graph Laplacian of patch similarities (isometry-invariant), and make patch features rotation-robust by max-aggregating patch features over four quarter-turn rotations (RFN).
Key results
- Spectral VMamba-B reaches 88.17% top-1 on miniImageNet, +1.00 over VMamba-B (87.17%) and +1.37 over XCiT-M24 (86.80%).
- Ours-T achieves 87.86% top-1 (+1.61 over VMamba-T 86.25%) with 3.9G FLOPs vs 4.9G for VMamba-T.
- Under rotation tests from 0 to 360 degrees, the method stays near 78% at non-canonical angles and near 87% at quarter-turn angles, while VMamba drops to about 50% at some angles.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace VMamba scanning with spectral traversal | VMamba-B · miniImageNet | Top-1 accuracy: 87.17 -> 88.17 | Helped |
| add the Rotational Feature Normalizer (RFN) (ablation) | Spectral VMamba · miniImageNet test images at canonical rotation angles | Top-1 accuracy | Helped |
| remove the Rotational Feature Normalizer (RFN) (ablation) | Spectral VMamba · miniImageNet test images across rotation angles | Top-1 accuracy | Hurt |
| add RFN without spectral traversal (ablation) | VMamba · miniImageNet test images across rotation angles | Top-1 accuracy | No clear effect |
| add spectral traversal (ablation) | VMamba with RFN · miniImageNet test images across rotation angles | Top-1 accuracy | Helped |
| increase the number of eigenvectors from 1 to 4 (ablation) | Spectral VMamba · miniImageNet | Top-1 accuracy: 86.72% -> 87.48% | Helped |
| vary nearest-neighbor count from k=5 to k=25 (ablation) | Spectral VMamba · miniImageNet | Top-1 accuracy: 87.48 -> 87.15 | No clear effect |
Surprising findings
- Spectral traversal alone without RFN still outperforms VMamba for rotations between 60 and 300 degrees.
- Adding RFN to VMamba alone does not improve rotational robustness, indicating patch-level invariance is insufficient without spectral traversal.
Limitations
- Not evaluated on medical or other domains; generalization not addressed.
- Rotation invariance is exact only for canonical quarter-turn angles; non-canonical angles stabilize around 78% accuracy.
- Assumes connected graph; disconnected components (e.g., in medical images) not handled.
Experimental setup
Datasets: miniImageNet
Benchmarks: miniImageNet
Metrics: top-1 accuracy top-5 accuracy FLOPs runtime memory usage
Baselines: VMamba MSVMamba LocalVim Vim Swin Transformer XCiT DeiT
Builds on: VMamba Mamba S4 spectral clustering normalized cuts
Closest papers by meaning
- Multi-Scale VMamba: Hierarchy in Hierarchy Visual State Space Model (NeurIPS 2024 · similarity 0.88)
- Spatial-Mamba: Effective Visual State Space Models via Structure-Aware State Fusion (ICLR 2025 · similarity 0.87)
- VMamba: Visual State Space Model (NeurIPS 2024 · similarity 0.86)
- S2MAE: A Spatial-Spectral Pretraining Foundation Model for Spectral Remote Sensing Data (CVPR 2024 · similarity 0.85)
- Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model (ICML 2024 · similarity 0.85)
- GroupMamba: Efficient Group-Based Visual State Space Model (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.