Research map / Architectures and efficiency / Attention and transformers
Separating the Wheat from the Chaff: Spatio-Temporal Transformer with View-interweaved Attention for Photon-Efficient Depth Sensing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SPDS-STFormer, a 3D spatio-temporal transformer with view-interweaved attention and adaptive weighting for photon-efficient SPAD depth sensing under low photon counts and low SBR.
Problem
Depth reconstruction from SPAD single-photon histograms under low photon counts and low signal-to-background ratio (SBR), where signal photons are swamped by background noise.
Approach
A transformer block that runs spatial-view multi-head self-attention and temporal-view global-guided self-attention in parallel and fuses them with a learned, SBR-aware adaptive weight (computed via channel and spatio-temporal attention plus channel shuffle), so the balance between views adapts to signal-background level.
Key results
- On the simulated Middlebury benchmark across nine SBR settings, SPDS-STFormer achieves the lowest RMSE in all settings, e.g., 0.0543 at 10:2 versus 0.0634 for Peng et al. 2022 and 0.0711 at 2:50 versus 1.7969 for Peng et al. 2022.
- On the self-collected real-world dataset (35 pairs, 4 ps bins), it reaches RMSE 9.59 mm versus 11.92 mm for Peng et al. 2022.
- Ablation shows removing adaptive weighting raises RMSE at 2:50 from 0.0711 to 0.0714 and fixing alpha degrades performance more.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SPDS-STFormer instead of the strongest listed baseline, Rapp et al. [2017] | SPAD depth sensing · Middlebury, 10:2 | RMSE: 0.0579 -> 0.0543 | Helped |
| Use SPDS-STFormer instead of the strongest listed baseline, Rapp et al. [2017] | SPAD depth sensing · Middlebury, 5:2 | RMSE: 0.0629 -> 0.0551 | Helped |
| Use SPDS-STFormer instead of the strongest listed baseline, Shin et al. [2016] | SPAD depth sensing · Middlebury, 2:2 | RMSE: 0.0668 -> 0.0607 | Helped |
| Use SPDS-STFormer instead of the strongest listed baseline, Rapp et al. [2017] | SPAD depth sensing · Middlebury, 10:10 | RMSE: 0.0527 -> 0.0481 | Helped |
| Use SPDS-STFormer instead of the strongest listed baseline, Rapp et al. [2017] | SPAD depth sensing · Middlebury, 5:10 | RMSE: 0.0628 -> 0.0607 | Helped |
| Use SPDS-STFormer instead of the strongest listed baseline, Rapp et al. [2017] | SPAD depth sensing · Middlebury, 2:10 | RMSE: 0.0602 -> 0.0581 | Helped |
| Use SPDS-STFormer instead of the strongest listed baseline, Rapp et al. [2017] | SPAD depth sensing · Middlebury, 10:50 | RMSE: 0.0555 -> 0.0516 | Helped |
| Use SPDS-STFormer instead of the strongest listed baseline, Rapp et al. [2017] | SPAD depth sensing · Middlebury, 5:50 | RMSE: 0.0566 -> 0.0528 | Helped |
| Add SPDS-STFormer instead of Peng et al. [2022] | Real-world SPAD depth sensing · Real-Captured Dataset | RMSE [mm]: 11.92 -> 9.59 | Helped |
| Add temporal-view global-guided self-attention (ablation) | SPDS-STFormer · Middlebury, high SBR (10:2) | RMSE: 0.0919 -> 0.0543 | Helped |
| Add temporal-view global-guided self-attention (ablation) | SPDS-STFormer · Middlebury, low SBR (2:50) | RMSE: 0.5005 -> 0.0711 | Helped |
| Add spatial-view multi-head self-attention (ablation) | SPDS-STFormer · Middlebury, high SBR (10:2) | RMSE: 0.5557 -> 0.0543 | Helped |
Surprising findings
- Spatial-only attention performs well at high SBR but produces checker artifacts at low SBR, while temporal-only attention is the reverse, motivating adaptive fusion.
Limitations
- Real-world evaluation uses only a self-collected dataset of 35 pairs captured with one prototype system.
- Training relies on simulated SPAD measurements from NYU v2 rather than real captured data.
Experimental setup
Datasets: NYU v2 (simulated SPAD histograms) Middlebury stereo dataset (simulated test) Self-collected real-world SPAD dataset (35 pairs)
Benchmarks: Middlebury (simulated SPAD) Self-collected real-world SPAD dataset
Metrics: RMSE
Baselines: LM Filter Shin et al. 2016 Rapp and Goyal 2017 Lindell et al. 2018 Peng et al. 2022
Builds on: ViT Non-local blocks (Peng et al. 2022) SM-FFN Squeeze-and-excitation (channel recalibration) ShuffleNet channel shuffle
Closest papers by meaning
- Learnable Burst-Encodable Time-of-Flight Imaging for High-Fidelity Long-Distance Depth Sensing (NeurIPS 2025 · similarity 0.85)
- Opportunistic Single-Photon Time of Flight (CVPR 2025 · similarity 0.83)
- Towards 3D Vision with Low-Cost Single-Photon Cameras (CVPR 2024 · similarity 0.82)
- Generalized Event Cameras (CVPR 2024 · similarity 0.82)
- Snapshot Lidar: Fourier Embedding of Amplitude and Phase for Single-Image Depth Reconstruction (CVPR 2024 · similarity 0.82)
- bit2bit: 1-bit quanta video reconstruction via self-supervised photon prediction (NeurIPS 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.