Research map / 3D vision / Gaussian splatting
USP-Gaussian: Unifying Spike-based Image Reconstruction, Pose Correction and Gaussian Splatting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes USP-Gaussian, an end-to-end framework that jointly optimizes a spike-to-image reconstruction network, spike-camera pose trajectories, and 3D Gaussian Splatting, using a self-supervised multi-reblur loss and a flip-and-minimum joint loss.
Problem
Cascaded spike-camera pipelines (spike-to-image reconstruction, then pose estimation, then novel view synthesis) accumulate errors, since poor reconstructed images degrade pose estimation and hence 3D reconstruction fidelity.
Approach
Jointly train Recon-Net, learnable start/end camera poses (SE(3) interpolation over the exposure interval), and 3DGS, with Recon-Net supervised self-supervisedly by a multi-reblur loss against the long-exposure image and a joint consistency loss between 3DGS and Recon-Net sequences that uses a flip-and-minimum operation to handle temporal order ambiguity.
Key results
- On the synthetic dataset, USP-Gaussian averages 27.903 PSNR / 0.843 SSIM / 0.217 LPIPS, versus 27.196 PSNR for SpikeGS.
- With 30% pose perturbation on the Wine scene, USP-Gaussian keeps 23.46 PSNR while SpikeGS drops to 16.44.
- Pose optimization reduces average translation error by 51.3% and rotation error by 72.4% relative to initial noisy poses.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the cascading pipeline with joint optimization of Recon-Net, camera poses, and 3DGS | USP-Gaussian · Synthetic dataset, Avg | PSNR: 27.196 -> 27.903 | Helped |
| add joint optimization of Recon-Net and 3DGS (ablation) | USP-Gaussian · Synthetic dataset, 3DGS outputs | PSNR: 22.392 -> 26.476 | Helped |
| add joint optimization of Recon-Net and 3DGS (ablation) | USP-Gaussian · Synthetic dataset, Recon-Net outputs | PSNR: 24.567 -> 27.138 | Helped |
| replace single-reblur loss with multi-reblur loss (ablation) | USP-Gaussian · Synthetic dataset, 3DGS outputs | PSNR: 25.659 -> 26.476 | Helped |
| replace single-reblur loss with multi-reblur loss (ablation) | USP-Gaussian · Synthetic dataset, Recon-Net outputs | PSNR: 25.571 -> 27.138 | Helped |
| add long-spike input alongside short-spike input (ablation) | Recon-Net · Synthetic dataset, spike-to-image reconstruction | SSIM: 0.859 -> 0.887 | Helped |
| add long-spike input alongside short-spike input (ablation) | Recon-Net · Synthetic dataset, spike-to-image reconstruction | LPIPS: 0.151 -> 0.136 | Helped |
| optimize camera poses jointly | USP-Gaussian · Wine, 10% pose perturbation | PSNR: 18.83 -> 26.05 | Helped |
| optimize camera poses jointly | USP-Gaussian · Wine, 30% pose perturbation | PSNR: 16.44 -> 23.46 | Helped |
Surprising findings
- Independent Recon-Net training tends to overfit due to the simplicity of the spike reconstruction task, and joint training improves both branches.
Limitations
- Requires longer training time than previous methods.
- Requires more GPU memory due to additional poses and Recon-Net parameters to optimize.
Experimental setup
Datasets: Deblur-NeRF scenes (synthetic spike simulation) self-collected real-world spike dataset
Benchmarks: Deblur-NeRF-based synthetic spike dataset (Wine, Tanabata, Factory, Outdoorpool) real-world spike dataset
Metrics: PSNR SSIM LPIPS translation error rotation error
Baselines: TFP-3DGS TFI-3DGS Spk2Img-3DGS SpikeNeRF SpikeGS
Builds on: 3D Gaussian Splatting BAD-Gaussian BAD-NeRF Blind Spot Network TFP
Closest papers by meaning
- SpikeGS: Reconstruct 3D Scene Captured by a Fast-Moving Bio-Inspired Camera (AAAI 2025 · similarity 0.90)
- COLMAP-Free 3D Gaussian Splatting (CVPR 2024 · similarity 0.89)
- SpikeNeRF: Learning Neural Radiance Fields from Continuous Spike Stream (CVPR 2024 · similarity 0.88)
- SfM-Free 3D Gaussian Splatting via Hierarchical Training (CVPR 2025 · similarity 0.87)
- Spatio-Temporal Interactive Learning for Efficient Image Reconstruction of Spiking Cameras (NeurIPS 2024 · similarity 0.87)
- Rethinking High-speed Image Reconstruction Framework with Spike Camera (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.