Research map / Architectures and efficiency / Attention and transformers

LESSViT: Robust Hyperspectral Representation Learning under Spectral Configuration Shift

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Efficient image transformers · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LESSViT, a channel-agnostic spatial-spectral ViT using low-rank LESS Attention, wavelength-aware SSRoPE, and a HyperMAE pretraining scheme, evaluated for robustness to spectral configuration shifts on SpectralEarth.

Problem

Hyperspectral models trained under one fixed spectral configuration (wavelength coverage, band sampling, channel count) fail to generalize across sensors, and explicit spatial-spectral attention in ViTs is computationally prohibitive at O(N^2 C^2).

Approach

LESS Attention factorizes joint spatial-spectral attention into separate spatial and spectral attention maps combined via a rank-r Kronecker-product sum, reducing complexity to O(rNC); paired with a tied channel-agnostic patch embedding, wavelength-based axial RoPE (SSRoPE), and a masked-autoencoder pretraining with decoupled masking and hierarchical channel sampling.

Key results

• LESSViT achieves the lowest average relative drop across spectral-shift settings (15% on C120_SW+, 26% on C82, 5% on C202 for the reported averages), and best on 14 of 15 OOD task-configuration pairs per the authors.
• On in-distribution CDL, LESSViT scores 51.78 mIoU versus SpectralViT's 70.29, so SpectralViT is stronger in-distribution.
• LESSViT's inference latency stays near-constant as channel count grows, while ChannelViT runs out of memory at C=200 on a 144 GB GPU.

Surprising findings

• SpectralViT is strongest in-distribution but degrades severely under spectral shift (e.g., CDL drops from 70.29 to 13.71 mIoU on C120_SW+).
• Increasing the HCS retained ratio from [0.2,0.3] to [0.4,0.5] changes performance only marginally while reducing pretraining time from 41 to 34 hours.

Limitations

• Generalization to entirely unseen or highly sparse spectral configurations remains challenging.
• Smaller patch sizes for better segmentation increase token count and computation; in-distribution performance trails SpectralViT.
• Evaluation limited to geospatial hyperspectral data.

Experimental setup

Datasets: SpectralEarth EnMAP

Benchmarks: SpectralEarth CDL SpectralEarth EuroCrops SpectralEarth CORINE SpectralEarth BDFORET SpectralEarth BNETD

Metrics: mIoU mAP relative drop normalized latency FLOPs

Baselines: SpectralViT DOFA HyperSigma ChannelViT SIREN

Builds on: Vision Transformer MAE RoPE ChannelViT LoRA

Closest papers by meaning

• S2MAE: A Spatial-Spectral Pretraining Foundation Model for Spectral Remote Sensing Data (CVPR 2024 · similarity 0.88)
• Hyperspectral Image Reconstruction via Combinatorial Embedding of Cross-Channel Spatio-Spectral Clues (AAAI 2024 · similarity 0.88)
• SPECAT: SPatial-spEctral Cumulative-Attention Transformer for High-Resolution Hyperspectral Image Reconstruction (CVPR 2024 · similarity 0.87)
• EigenSR: Eigenimage-Bridged Pre-Trained RGB Learners for Single Hyperspectral Image Super-Resolution (AAAI 2025 · similarity 0.86)
• HyperFree: A Channel-adaptive and Tuning-free Foundation Model for Hyperspectral Remote Sensing Imagery (CVPR 2025 · similarity 0.86)
• VolFormer: Explore More Comprehensive Cube Interaction for Hyperspectral Image Restoration and Beyond (CVPR 2025 · similarity 0.85)