Research map / Video / Video segmentation and tracking
Toward Dynamic Non-Line-of-Sight Imaging with Mamba Enforced Temporal Consistency
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ST-Mamba, a spatial-temporal Mamba network that fuses neighbouring transient frames from under-scanned confocal NLOS measurements to reconstruct dynamic hidden volumes, trained with a phasor-field wave loss on a new synthetic dynamic NLOS dataset and tested on self-captured real videos.
Problem
Dynamic reconstruction of hidden 3D volumes in confocal non-line-of-sight imaging is limited by slow dense raster scanning, and existing under-scanning methods process single frames and ignore temporal consistency across transient frames.
Approach
Spatial-temporal Mamba blocks exploit the causality along the time axis of transient histograms and bidirectional spatial SSMs, a cross ST-Mamba gates adjacent-frame features into the reference frame, and a phasor-field wave loss is applied to suppress noise in low-SNR transients.
Key results
- On the synthetic test set, the multi-frame model reaches 24.46 PSNR / 84.08 SSIM for intensity and 0.0880 RMSE / 0.0397 MAD for depth, beating the best baseline USM (23.80 / 80.85 / 0.0945 / 0.0432).
- The single-frame variant Ours-S (23.97 PSNR) also outperforms all baselines, and the multi-frame model improves over it on all metrics.
- Inference takes 0.359 s per volume with about 17 GB memory, versus 0.149 s for USM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multi-frame input and cross ST-Mamba instead of the single-frame Ours-S | Ours · Synthetic test data | PSNR: 23.97 -> 24.46 | Helped |
| Add phasor field loss Lpf (ablation) | Ours · Synthetic test data | PSNR: 24.19 -> 24.47 | Helped |
| Add regularized loss Lls,tv (ablation) | Ours · Synthetic test data | PSNR: 24.47 -> 24.46 | Hurt |
| Add regularized loss Lls,tv (ablation) | Ours · Synthetic test data | SSIM: 83.07 -> 84.08 | Helped |
| Add regularized loss Lls,tv (ablation) | Ours · Synthetic test data | RMSE: 0.0905 -> 0.0880 | Helped |
| Add regularized loss Lls,tv (ablation) | Ours · Synthetic test data | MAD: 0.0404 -> 0.0397 | Helped |
| Use both spatial S-Mamba and temporal T-Mamba (ablation) | Ours · Synthetic test data | PSNR: 24.38 -> 24.46 | Helped |
| Use both spatial S-Mamba and temporal T-Mamba rather than spatial Mamba only (ablation) | Ours · Synthetic test data | MAD: 0.0496 -> 0.0397 | Helped |
| Enable causal operation (ablation) | Mamba encoder and Mamba fusion · Synthetic test data | PSNR: 23.94 -> 24.46 | Helped |
| Replace ViT encoder with Mamba encoder (ablation) | Mamba encoder, Mamba fusion · Synthetic test data | PSNR: 23.77 -> 24.46 | Helped |
| Replace ViT fusion with Mamba fusion (ablation) | Mamba encoder, Mamba fusion · Synthetic test data | PSNR: 24.32 -> 24.46 | Helped |
| Add Cross ST-Mamba fusion while retaining the same number of SSMs (ablation) | Mamba encoder · Synthetic test data | PSNR: 23.97 -> 24.46 | Helped |
Surprising findings
- Replacing Cross ST-Mamba with cross-attention lowers performance, suggesting gated Mamba fusion suits partially causal transient data better than attention.
Limitations
- Multi-frame fusion increases inference time substantially.
- The Fourier-based feature transformation relies on the PSF and is computationally expensive; the authors call for a lightweight alternative.
- Evaluation on real data uses only 4 sequences captured by a single prototype.
Experimental setup
Datasets: synthetic dynamic NLOS dataset (265 sequences) real-world dynamic NLOS videos (4 sequences)
Benchmarks: synthetic dynamic NLOS test set (17 sequences) real-world captured NLOS videos (4 sequences)
Metrics: PSNR SSIM RMSE MAD
Baselines: LCT RSD FK LFE I-K CSA USM Ours-S
Builds on: Mamba bidirectional SSM (Vim) USM phasor field waves LFE
Closest papers by meaning
- Virtual Scanning: Unsupervised Non-line-of-sight Imaging from Irregularly Undersampled Transients (NeurIPS 2024 · similarity 0.84)
- MambaSCI: Efficient Mamba-UNet for Quad-Bayer Patterned Video Snapshot Compressive Imaging (NeurIPS 2024 · similarity 0.84)
- LC-Mamba: Local and Continuous Mamba with Shifted Windows for Frame Interpolation (CVPR 2025 · similarity 0.84)
- EventMamba: Enhancing Spatio-Temporal Locality with State Space Models for Event-Based Video Reconstruction (AAAI 2025 · similarity 0.83)
- Spatial-Mamba: Effective Visual State Space Models via Structure-Aware State Fusion (ICLR 2025 · similarity 0.83)
- Sp3ctralMamba: Physics-Driven Joint State Space Model for Hyperspectral Image Reconstruction (AAAI 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.