Research map / 3D vision / Depth and stereo
In2SET: Intra-Inter Similarity Exploiting Transformer for Dual-Camera Compressive Hyperspectral Imaging
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes In2SET, a Transformer denoiser with intra-similarity (PAN-derived spatial attention plus channel self-attention) and inter-similarity (cosine-reweighted cross-modal attention) inside a PAN-guided deep unrolling framework for dual-camera compressive hyperspectral reconstruction.
Problem
Reconstructing a 3D hyperspectral image from a dual-camera compressive hyperspectral imaging (DCCHI) system, fusing a CASSI compressive measurement with a panchromatic (PAN) image, where the HSI prior is ill-posed and existing PAN-guided methods rely on hand-crafted priors or single-perspective cros
Approach
Use the intra-spatial similarity of the PAN image as a proxy for the unavailable HSI intra-similarity, and use cosine-similarity reweighting between HSI and PAN semantic features as inter-similarity attention, all embedded as the denoiser in a PAN-guided half-quadratic-splitting unrolling network.
Key results
- In2SET-9stg achieves average PSNR 43.69 dB and SSIM 0.990 on 10 simulated KAIST scenes, the best among compared methods.
- In2SET-3stg reaches performance comparable to DAUHST-DC-5stg at 20.78 GFLOPs versus 40.50 GFLOPs.
- Ablation: adding CRW raises PSNR from 37.23 to 40.05 dB on the 2-stage baseline; MHA-C and MHA-S add a further +0.41 and +0.38 dB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use In2SET-9stg instead of DAUHST-DC-9stg | DCCHI reconstruction · KAIST simulated, average | PSNR: 42.62 -> 43.69 | Helped |
| Use In2SET-9stg instead of DAUHST-DC-9stg | DCCHI reconstruction · KAIST simulated, average | GFLOPs: 72.11 -> 59.40 | Hurt |
| Add CRW (ablation) | In2SET baseline · KAIST simulated, average | PSNR: 37.23 -> 40.05 (+2.82 dB) | Helped |
| Add MHA-C to CRW (ablation) | In2SET · KAIST simulated, average | PSNR: +0.41 dB | Helped |
| Add MHA-S to CRW + MHA-C (ablation) | In2SET · KAIST simulated, average | PSNR: +0.38 dB | Helped |
| Increase CG iterations from 1 to 10 (ablation) | In2SET data-fidelity solver · KAIST simulated, average | PSNR: 40.42 -> 41.05 | Helped |
| Increase CG iterations from 1 to 10 (ablation) | In2SET data-fidelity solver · KAIST simulated, average | FPS: 26.45 -> 13.99 | Hurt |
Limitations
- Relies on handcrafted ablation of CG iterations; more stages increase FLOPs substantially (59.40 GFLOPs at 9 stages)
- Intra-similarity proxy from PAN is an approximation, validated only via correlation maps on public datasets
- Real-data evaluation is qualitative on a single scene
Experimental setup
Datasets: CAVE KAIST ICVL
Benchmarks: KAIST 10-scene simulated test set real DCCHI measurement Ninja
Metrics: PSNR SSIM GFLOPs FPS
Baselines: PFusion-RGB PIDS-RGB TV-DC PIDS-DC BiSRNet-DC CST-DC HDNet-DC MST-DC MST++-DC DAUHST-DC
Builds on: MST DAUHST half quadratic splitting CASSI PIDS
Closest papers by meaning
- Dual-Window Multiscale Transformer for Hyperspectral Snapshot Compressive Imaging (AAAI 2024 · similarity 0.87)
- SPECAT: SPatial-spEctral Cumulative-Attention Transformer for High-Resolution Hyperspectral Image Reconstruction (CVPR 2024 · similarity 0.86)
- Improving Spectral Snapshot Reconstruction with Spectral-Spatial Rectification (CVPR 2024 · similarity 0.85)
- Spectral Compressive Imaging via Chromaticity-Intensity Decomposition (NeurIPS 2025 · similarity 0.85)
- Learning Multi-Modal Cross-Scale Deformable Transformer Network for Unregistered Hyperspectral Image Super-resolution (AAAI 2024 · similarity 0.85)
- DiffSCI: Zero-Shot Snapshot Compressive Imaging via Iterative Spectral Diffusion Model (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.