Research map / Video / Video segmentation and tracking
bit2bit: 1-bit quanta video reconstruction via self-supervised photon prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes bit2bit, a self-supervised method that reconstructs dense video from 1-bit SPAD quanta data by splitting binary photon events into input/target pairs and applying a masked cross-entropy loss.
Problem
Reconstructing clean, high-quality image stacks at full spatiotemporal resolution from sparse 1-bit quanta image (SPAD) data, where binning degrades temporal resolution and standard self-supervised denoisers (GAP, Noise2Noise) fail because binary photon events are not Poisson-distributed.
Approach
Randomly thin binary photon events into input and target volumes and apply a loss masked at input-1 locations, removing the complementary dependency between input and target that causes artifacts in binary data; combined with 3D ResUNet, random-p sampling, and group normalization.
Key results
- bit2bit with masked loss reaches 33.93 PSNR / 0.959 SSIM on simulated data, versus 20.89 without masking.
- Ensembling 10 Bernoulli-sampled inferences with p in [0.3,0.7] raises PSNR to 34.35.
- Standard GAP 2D with mask reaches 29.09 PSNR; supervised upper bound is 36.51 PSNR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply masked loss to the split input and target (ablation) | bit2bit · Simulated video | PSNR: 20.89/0.84 -> 33.93/0.99 | Helped |
| Apply masked loss to the split input and target (ablation) | N2N · Simulated video | PSNR: 23.14/0.73 -> 28.83/0.58 | Helped |
| Apply masked loss (ablation) | GAP · Simulated video | PSNR: 20.54/0.64 -> 29.09/0.57 | Helped |
| Extend GAP-style reconstruction from 2D to 3D | bit2bit · Simulated video | PSNR: 29.09/0.57 -> 33.93/0.99 | Helped |
| Use group normalization size 8 instead of size 1 (ablation) | bit2bit · Simulated video | PSNR: 30.85 -> 33.93 | Helped |
| Raise the lower bound of the p sampling range (ablation) | bit2bit · Simulated video | PSNR: 33.93 -> 32.05 | Hurt |
| Lower the upper bound of the p sampling range to 1 - 1E-2 (ablation) | bit2bit · Simulated video | PSNR: 33.93 -> 32.62 | Hurt |
| Use fixed p near 1 instead of stochastic p sampling (ablation) | bit2bit · Simulated video | PSNR: 33.93 -> 26.12 | Hurt |
| Increase startup filter size from 24 to 48 (ablation) | bit2bit · Simulated video | PSNR: 33.09 -> 32.17 | Hurt |
| Use 10-shot inference rather than one-shot inference (ablation) | bit2bit, p range [0.3, 0.7] · Simulated video | PSNR: 34.08 -> 34.35 | Helped |
| Train with p range [0.3, 0.7] instead of [0.5, 0.5] for 10-shot inference (ablation) | bit2bit · Simulated video | PSNR: 34.00 -> 34.35 | Helped |
Surprising findings
- Without the masked loss, the network learns a deterministic input-target relation and produces dark artifacts at input photon locations, so naive N2N/GAP splitting fails on binary data.
- Larger models (e.g., 48 start filters) performed worse than 32 or 24, suggesting overfitting.
Limitations
- Requires offline training per dataset; model should be trained on similar scenes.
- Assumes a simple Poisson point process model and does not handle other noise types.
- Lack of ground truth for real data limits quantitative evaluation; simulated results are specific to the chosen configuration.
Experimental setup
Datasets: Simulated 1-bit data from iPhone 15 240 fps slow-motion video Real SPAD512S videos (7 scenes) Real SPAD video from QBP work
Benchmarks: Simulated 1-bit video from iPhone slow-motion reference
Metrics: PSNR SSIM
Baselines: Supervised training Noise2Noise Noise2Void GAP Quanta Burst Photography (QBP)
Builds on: Generative Accumulation of Photons (GAP) Noise2Noise Noise2Void Noise2Self 3D U-Net
Closest papers by meaning
- Generative Quanta Color Imaging (CVPR 2024 · similarity 0.83)
- Image Reconstruction from Readout-Multiplexed Single-Photon Detector Arrays (CVPR 2025 · similarity 0.83)
- 2DQuant: Low-bit Post-Training Quantization for Image Super-Resolution (NeurIPS 2024 · similarity 0.83)
- Learning Efficient Fuse-and-Refine for Feed-Forward 3D Gaussian Splatting (NeurIPS 2025 · similarity 0.83)
- Image and Video Tokenization with Binary Spherical Quantization (ICLR 2025 · similarity 0.82)
- MambaSCI: Efficient Mamba-UNet for Quad-Bayer Patterned Video Snapshot Compressive Imaging (NeurIPS 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.