Research map / 3D vision / Depth and stereo
From Pixels to Views: Learning Angular-Aware and Physics-Consistent Representations for Light Field Microscopy
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the XLFM-Zebrafish benchmark and XLFM-Former, a Swin Transformer encoder-CNN decoder for XLFM 3D reconstruction, with view-masked pretraining (MVM-LF) and a PSF-based optical rendering consistency loss (ORC).
Problem
Learning-based 3D reconstruction from extended light field microscopy (XLFM) data is underdeveloped due to a lack of standardized datasets and methods that model angular-spatial structure while staying physically grounded.
Approach
Treat sub-aperture views as the atomic modeling unit: pretrain a Swin Transformer encoder by masking 70% of views and reconstructing them, and add a loss that compares PSF forward projections of predicted and ground-truth volumes, so the network is constrained by the optical forward model.
Key results
- Full XLFM-Former reaches 54.04 dB PSNR and 0.9944 SSIM on XLFM-Zebrafish average, vs 50.16 dB for the best baseline ConvNeXt.
- MVM-LF pretraining with 70% view masking gives 53.38 dB vs 52.14 dB from scratch; ORC loss alone gives 52.96 dB.
- On H2B-Nemos, XLFM-Former gains +0.92 dB PSNR over ResNet-101 in the supervised setting and +2.29 dB in zero-shot.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add XLFM-Former instead of ConvNeXt | XLFM-Zebrafish reconstruction · XLFM-Zebrafish test set (average) | PSNR: 50.16 -> 54.04 | Helped |
| Add ORC loss only (ablation) | XLFM-Former · XLFM-Zebrafish | PSNR: 52.14 -> 52.96 | Helped |
| Add MVM-LF pretraining only (ablation) | XLFM-Former · XLFM-Zebrafish | PSNR: 52.14 -> 53.38 | Helped |
| Use MVM-LF pretraining rather than train from scratch (ablation) | XLFM-Former · XLFM-Zebrafish, 10% labeled data | PSNR: 50.73 -> 51.92 | Helped |
| Increase masking ratio to 90% instead of 70% (ablation) | MVM-LF pretraining · XLFM-Zebrafish reconstruction | PSNR: 53.38 -> 52.30 | Hurt |
| Add view-aware masking rather than pixel-random masking (ablation) | XLFM-Former pretraining · XLFM-Zebrafish | PSNR: 52.97 -> 54.04 | Helped |
| Use only 80% of input views with MVM-LF pretraining (ablation) | XLFM-Former · XLFM-Zebrafish | PSNR: 52.14 -> 53.26 | Helped |
| Add MVM-LF pretraining only (ablation) | XLFM-Former · H2B-NeMOs | PSNR: 50.87 -> 51.66 | Helped |
| Add ORC loss only (ablation) | XLFM-Former · H2B-NeMOs | PSNR: 50.87 -> 51.77 | Helped |
| Increase PSF FWHM by 10% (ablation) | ORC loss in XLFM-Former · H2B-NeMOs | PSNR: 52.3440 -> 52.3369 | No clear effect |
| Decrease PSF FWHM by 10% (ablation) | ORC loss in XLFM-Former · H2B-NeMOs | PSNR: 52.3440 -> 52.4647 | No clear effect |
| Add MS-SSIM loss (ablation) | XLFM-Former reconstruction loss · XLFM-Zebrafish | PSNR: 53.2787 -> 54.0435 | Helped |
Surprising findings
- ImageNet-1k/22k pretraining gives only marginal gains (52.70/52.38 dB), and MAE with a ViT backbone performs much worse (46.55 dB) than scratch training.
Limitations
- Evaluated only on zebrafish datasets, limiting generalization to other organisms such as mouse or drosophila.
- Reports neural activity reconstruction quality but does not assess functional trace extraction.
- ORC loss uses the PSF-based projection of the ground-truth volume rather than the measured light field, so it does not directly use measurements.
Experimental setup
Datasets: XLFM-Zebrafish H2B-Nemos
Benchmarks: XLFM-Zebrafish H2B-Nemos
Metrics: PSNR SSIM
Baselines: ConvNeXt ViT PVT EfficientNet ResNet-50 ResNet-101 U-Net MAE ImageNet pretraining
Builds on: Swin Transformer masked autoencoders U-Net
Closest papers by meaning
- V2V3D: View-to-View Denoised 3D Reconstruction for Light Field Microscopy (CVPR 2025 · similarity 0.90)
- NeLF-Pro: Neural Light Field Probes for Multi-Scale Novel View Synthesis (CVPR 2024 · similarity 0.86)
- S2D-LFE: Sparse-to-Dense Light Field Event Generation (CVPR 2025 · similarity 0.85)
- ProbeSDF: Light Field Probes For Neural Surface Reconstruction (CVPR 2025 · similarity 0.85)
- CD-RCM: Generalizable Continuous-Depth Novel View Synthesis for Reflectance Confocal Microscopy (NeurIPS 2026 · similarity 0.85)
- Occlusion-Embedded Hybrid Transformer for Light Field Super-Resolution (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.