Research map / Image generation / Super-resolution
Learning Multi-Modal Cross-Scale Deformable Transformer Network for Unregistered Hyperspectral Image Super-resolution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes M2DTN, a multi-modal cross-scale deformable transformer that jointly registers and fuses unregistered low-resolution hyperspectral and high-resolution multispectral images for hyperspectral super-resolution.
Problem
Fusion-based hyperspectral image super-resolution assumes the auxiliary HrMSI is strictly registered with the LrHSI, which fails in practice due to platform, viewpoint and atmospheric distortions; CNN-based fusion also lacks global context.
Approach
Formulates a spectrum-preserving spatial-guided registration-SR model (SSRU) and maps it to a network with a multi-modal registration deformable module (MMRD) that predicts deformation fields from cross-attention spatial relations, a multi-scale spectral-attention transformer (MSFT) for feature extraction, and a cross-scale feature aggregation module (CSFA) for reconstruction.
Key results
- On Pavia Center, M2DTN reaches PSNR 45.3475 and SAM 2.7471, versus 41.0935 PSNR for the next best method u2MDN.
- On Houston, M2DTN reaches PSNR 48.8570 and SAM 1.0295, versus 42.6521 PSNR for u2MDN.
- On Harvard, M2DTN reaches PSNR 45.8886, versus 44.5377 for u2MDN.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use M2DTN instead of u2MDN | Unregistered HSI-SR · Pavia Center | PSNR: 41.0935 -> 45.3475 | Helped |
| use M2DTN instead of u2MDN | Unregistered HSI-SR · Houston | PSNR: 42.6521 -> 48.8570 | Helped |
| use M2DTN instead of u2MDN | Unregistered HSI-SR · Harvard | PSNR: 44.5377 -> 45.8886 | Helped |
| add MMRD (ablation) | M2DTN · Pavia Center | PSNR: 43.6667 -> 45.3475 | Helped |
| add MMRD (ablation) | M2DTN · Houston | PSNR: 45.2705 -> 48.8570 | Helped |
| add MMRD (ablation) | M2DTN · Harvard | PSNR: 43.0694 -> 45.8886 | Helped |
| use CSFA instead of a U-net decoder (ablation) | M2DTN · Pavia Center | PSNR: 41.6127 -> 45.3475 | Helped |
| use CSFA instead of a U-net decoder (ablation) | M2DTN · Houston | PSNR: 43.1989 -> 48.8570 | Helped |
| use CSFA instead of a U-net decoder (ablation) | M2DTN · Harvard | PSNR: 39.7439 -> 45.8886 | Helped |
| use global spectral feature attention instead of ordinary multi-head self-attention (ablation) | M2DTN · Pavia Center | PSNR: 36.4423 -> 45.3475 | Helped |
| use global spectral feature attention instead of ordinary multi-head self-attention (ablation) | M2DTN · Houston | PSNR: 32.5845 -> 48.8570 | Helped |
| use global spectral feature attention instead of ordinary multi-head self-attention (ablation) | M2DTN · Harvard | PSNR: 23.2044 -> 45.8886 | Helped |
Surprising findings
- Traditional methods such as CNMF, SFIM, GSA and Hysure perform better on the Harvard dataset than on the other datasets, indicating strong sensitivity to imaging conditions.
Limitations
- Relies on a simulated elastic-transformation distortion rather than real misregistered sensor pairs.
- Hyperformer, a registered-data method, fails on unregistered inputs, and the proposed method's gains are evaluated only on simulated data.
Experimental setup
Datasets: Pavia Center Houston Harvard
Benchmarks: Pavia Center Houston Harvard
Metrics: CC SAM RMSE ERGAS PSNR
Baselines: NonReg PixAwa CUCaNet u2MDN Hyperformer GSA Hysure CNMF SFIM
Builds on: Hyperformer VoxelMorph U-Net Spatial Transformer Network
Closest papers by meaning
- MFTN: A Multi-scale Feature Transfer Network Based on IMatchFormer for Hyperspectral Image Super-Resolution (ICML 2024 · similarity 0.89)
- EigenSR: Eigenimage-Bridged Pre-Trained RGB Learners for Single Hyperspectral Image Super-Resolution (AAAI 2025 · similarity 0.88)
- Dual-Window Multiscale Transformer for Hyperspectral Snapshot Compressive Imaging (AAAI 2024 · similarity 0.87)
- S2CycleDiff: Spatial-Spectral-Bilateral Cycle-Diffusion Framework for Hyperspectral Image Super-resolution (AAAI 2024 · similarity 0.87)
- Enhancing Hyperspectral Images via Diffusion Model and Group-Autoencoder Super-resolution Network (AAAI 2024 · similarity 0.86)
- Sp3ctralMamba: Physics-Driven Joint State Space Model for Hyperspectral Image Reconstruction (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.