Research map / Image generation / Image restoration
Illumination Spectrum Estimation for Multispectral Images via Surface Reflectance Modeling and Spatial-Spectral Feature Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ISS, a deep IRSS framework that combines a learnable GMM-based spectral unmixing block with a cross-attention feature mixing block (iterated with a GRU) to estimate the illumination spectrum of multispectral images.
Problem
Illumination spectrum estimation for multispectral images is hard because illuminant and surface reflectance spectra are entangled and spectral diversity is high; prior methods underuse joint reflectance modeling and spatial-spectral fusion.
Approach
Model surface reflectance with a learnable spectral unmixing block (GMM-initialized endmembers and abundances, learned via convolution/linear layers) and fuse its endmember features with CNN spatial features via cross-attention whose attention is biased by abundance maps, refined iteratively with a GRU, while jointly reconstructing reflectance and illumination.
Key results
- On KAUST, ISS achieves 5.55 mean-ΔAMS and 3.29 std-ΔAMS, versus 7.30 and 4.45 for BeyondRGB (best prior).
- On CAVE (unseen during training), ISS achieves 5.93 mean-ΔAMS vs 6.64 for BeyondRGB.
- On the BeyondRGB Lab and Field sets, ISS achieves 3.65 and 5.97 mean-ΔAMS, respectively; removing the FMB raises KAUST error to 8.05 from 4.40 in ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace BeyondRGB with ISS | illumination spectrum estimation · KAUST | mean-∆AMS: 7.30 -> 5.55 | Helped |
| Replace BeyondRGB with ISS | illumination spectrum estimation · CAVE | mean-∆AMS: 6.64 -> 5.93 | Helped |
| Replace BeyondRGB with ISS | illumination spectrum estimation · BeyondRGB Lab | mean-∆AMS: 5.92 -> 3.65 | Helped |
| Replace BeyondRGB with ISS | illumination spectrum estimation · BeyondRGB Field | mean-∆AMS: 7.22 -> 5.97 | Helped |
| Add the spectral unmixing (SU) block (ablation) | ISS · Ablation study | mean-∆AMS: 5.52 -> 4.40 | Helped |
| Add the feature mixing block (FMB) (ablation) | ISS · Ablation study | mean-∆AMS: 8.05 -> 4.40 | Helped |
| Add the spatial feature extractor (SFE) (ablation) | ISS · Ablation study | mean-∆AMS: 24.64 -> 4.40 | Helped |
| Add the reconstruction loss (LRC) (ablation) | ISS · Ablation study | mean-∆AMS: 4.72 -> 4.40 | Helped |
| Add the illumination estimation loss (LAE) (ablation) | ISS · Ablation study | mean-∆AMS: 73.59 -> 4.40 | Helped |
Surprising findings
- Removing the illumination angular-error loss increases mean-ΔAMS by about 73 degrees, showing reconstruction loss alone gives almost no illumination constraint.
Limitations
- Designed for a single global illumination spectrum; locally varying or multi-illuminant scenes are not handled.
- Relies on relighting synthetic data (KAUST/CAVE) for training and evaluation, with real-world evaluation limited to BeyondRGB.
Experimental setup
Datasets: KAUST CAVE BeyondRGB
Benchmarks: KAUST CAVE BeyondRGB
Metrics: mean-ΔAMS std-ΔAMS
Baselines: GrayEdge LRMF ISNL PWIR DUN BeyondRGB
Builds on: GMM-based spectral unmixing DUN BeyondRGB
Closest papers by meaning
- Spectral Compressive Imaging via Chromaticity-Intensity Decomposition (NeurIPS 2025 · similarity 0.84)
- Automatic Spectral Calibration of Hyperspectral Images: Method, Dataset and Benchmark (CVPR 2025 · similarity 0.84)
- Improving Spectral Snapshot Reconstruction with Spectral-Spatial Rectification (CVPR 2024 · similarity 0.83)
- Hyperspectral Image Reconstruction via Combinatorial Embedding of Cross-Channel Spatio-Spectral Clues (AAAI 2024 · similarity 0.83)
- Diffusion Reflectance Map: Single-Image Stochastic Inverse Rendering of Illumination and Reflectance (CVPR 2024 · similarity 0.83)
- FRN: Fractal-Based Recursive Spectral Reconstruction Network (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.