Research map / 3D vision / Depth and stereo
Real-time Acquisition and Reconstruction of Dynamic Volumes with Neural Structured Illumination
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a jointly optimized neural structured illumination and single-pixel 1D-decoder network for real-time acquisition and reconstruction of dynamic 3D density volumes at 40 volumes per second with a projector-camera setup.
Problem
High-fidelity, real-time acquisition and reconstruction of temporally-varying 3D density volumes (e.g., fluid phenomena) from sparse 2D camera measurements under limited sampling budget.
Approach
Jointly learns the structured light patterns (encoder weights, passed through sigmoid) with a per-pixel 1D decoder that predicts density along each camera ray from single-pixel measurements and resampled local incident lighting, followed by a 3D UNet aggregation across cameras, trained entirely on simulated fluid data.
Key results
- Achieves 40 volumes per second at 128^3 resolution using 3 cameras and 6 light patterns, with 9.2 ms reconstruction time per volume.
- On a real static object, the single-camera setup (#v=1, #p=12) reaches SSIM 0.90 / PSNR 22.31 versus 0.85 / 21.15 for compressive sensing [19] with one view.
- Synthetic smoke comparisons show superior quality to PINF and GlobalTrans, which take roughly 10 to 40 minutes per volume.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Jointly optimize the structured-light patterns with the reconstruction network instead of using fixed random binary or Hadamard patterns (ablation) | Neural structured illumination · Synthetic reconstruction with 1 camera | Reconstruction quality | Helped |
| Increase the number of light patterns (ablation) | Neural structured illumination · Validation data | Validation loss | Helped |
| Increase the number of cameras (ablation) | Neural structured illumination · Validation data | Validation loss | Helped |
| Use equally spaced view angles for cam1 and cam2 (ablation) | Multi-camera reconstruction · Synthetic reconstruction at volumetric resolution 323 | Reconstruction error | Helped |
| Replace local incident lighting input with camera index and screen-space coordinates (ablation) | Neural structured illumination decoder · Synthetic sample | Reconstruction accuracy | Hurt |
| Aggregate camera-space volumes instead of volumes in a common global space (ablation) | Multi-camera reconstruction · Synthetic sample | Reconstruction efficiency | Hurt |
| Replace the 3D UNet aggregation module with a 4-layer MLP (ablation) | Multi-camera reconstruction · Synthetic sample | Reconstruction quality | No clear effect |
| Replace the structured pipeline with an end-to-end network of 9 convolution layers (ablation) | Neural structured illumination · Synthetic sample | Reconstruction accuracy | Hurt |
| Use the proposed method instead of PINF or GlobalTrans | Real static object reconstruction · Real static object | PSNR | Helped |
| Use the proposed method instead of PINF or GlobalTrans | Synthetic smoke sequence reconstruction · Synthetic smoke sequence | Reconstruction errors | Helped |
Surprising findings
- The lowest reconstruction error for the cam1/cam2 view angles is achieved with all view angles equally split across pi, supporting the multi-cam configuration.
Limitations
- Assumes the physical phenomenon stays static during the multi-pattern projection, and no attenuation, refraction, reflection or multiple scattering.
- Each volume is reconstructed independently without exploiting temporal coherence.
- Requires a controlled projector-camera setup, unlike uncontrolled-illumination methods.
Experimental setup
Datasets: Mantaflow simulated fluid density volumes
Benchmarks: real static object (flower-like shape) comparison synthetic smoke sequence
Metrics: SSIM PSNR RMSE
Baselines: PINF GlobalTrans Kang et al. visible-light tomography Compressive sensing volumetric acquisition Optical sectioning (light slices)
Builds on: Mantaflow PINF Compressive sensing volumetric acquisition (Gu et al.) Light slice optical sectioning (Ihrke/Wu et al.)
Closest papers by meaning
- TurboSL: Dense Accurate and Fast 3D by Neural Inverse Structured Light (CVPR 2024 · similarity 0.87)
- Neural Inverse Rendering from Propagating Light (CVPR 2025 · similarity 0.85)
- Light Transport-aware Diffusion Posterior Sampling for Single-View Reconstruction of 3D Volumes (CVPR 2025 · similarity 0.85)
- Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting (ICLR 2024 · similarity 0.85)
- ProbeSDF: Light Field Probes For Neural Surface Reconstruction (CVPR 2025 · similarity 0.84)
- 3D Reconstruction with Generalizable Neural Fields using Scene Priors (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.