Research map / 3D vision / NeRF and radiance fields
MultimodalStudio: A Heterogeneous Sensor Dataset and Framework for Neural Rendering across Multiple Imaging Modalities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MultimodalStudio (MMS-DATA, a 32-scene 5-modality multi-view dataset, and MMS-FW, a modular multimodal NeRF framework) and shows that jointly training on RGB with other modalities improves rendering quality and enables cross-modality transfer.
Problem
Neural radiance fields are mostly trained on RGB, and the ability of implicit neural models to learn and transfer information across heterogeneous imaging modalities (NIR, mono, polarization, multispectral) has been little explored, largely due to scarce multimodal multi-view data.
Approach
A shared hash-grid NeRF with a shared density field and a multi-output radiance MLP, where each modality supervises only its own channels, trained directly on mosaicked raw frames from heterogeneous, independently calibrated sensors, so modalities transfer information to each other.
Key results
- Two-modality training raised RGB PSNR by roughly 2.5-3 dB over single-modality RGB training on mosaicked MMS-DATA (e.g., RGB-NIR: 31.00 vs 27.76 baseline gain of +3.24 dB).
- On the X-NeRF dataset MMS-FW reached 33.75 PSNR for RGB versus 31.77 for X-NeRF; on MMS-DATA it reached 32.42 vs 25.73 for X-NeRF.
- Training with only one RGB-paired second-modality view every two still beat single-modality training, and training on raw mosaicked frames performed comparably to demosaicked frames (e.g., MS 30.69 raw vs 30.05 demosaicked).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a second modality to RGB training | MMS-FW · MMS-DATA mosaicked; two-modality tests | PSNR: RGB rendering gains ∼2.5-3 dB; the second modality around ∼2.4 dB | Helped |
| Add a second modality while halving views and rays per iteration (ablation) | MMS-FW · MMS-DATA mosaicked; two-modality tests | PSNR | Helped |
| Add Pol to RGB training | MMS-FW · MMS-DATA mosaicked; RGB and Pol | PSNR: 28.83 (+1.25) | Helped |
| Train with RGB and fewer than 5 Pol frames (ablation) | MMS-FW · MMS-DATA; RGB rendering | PSNR | Hurt |
| Use raw mosaicked frames instead of demosaicked frames (ablation) | MMS-FW · MMS-DATA; RGB and Mono | PSNR: 32.34 vs 32.00; 32.43 vs 32.31 | Mixed |
| Use a smaller ray batch for demosaicked training (ablation) | MMS-FW · MMS-DATA demosaicked | PSNR | No clear effect |
| Supervise a smaller channel set per pixel during demosaicked training (ablation) | MMS-FW · MMS-DATA demosaicked | PSNR | No clear effect |
| Replace X-NeRF with MMS-FW | NeRF multimodal rendering · X-NeRF Dataset | PSNR: 31.77 -> 33.75; 31.60 -> 33.33; 33.22 -> 36.11 | Helped |
| Replace X-NeRF with MMS-FW | NeRF multimodal rendering · MMS-DATA | PSNR: 25.73 -> 32.42; 26.97 -> 32.97; 23.63 -> 29.89 | Helped |
Surprising findings
- Polarization frames sometimes hurt RGB rendering when fewer than about 5 Pol views are available, as the Pol constraints lead to density overfitting.
- Raw mosaicked training is roughly on par with demosaicked training, and is even better for some modalities such as MS in the five-modality setting.
Limitations
- Geometry evaluation is not possible numerically due to lack of ground-truth geometry.
- Polarization modality can degrade other modalities when few Pol views are available.
- Scenes were captured with manually moved rig, so sensor poses vary across scenes.
Experimental setup
Datasets: MMS-DATA X-NeRF dataset
Benchmarks: MMS-DATA X-NeRF dataset
Metrics: PSNR SSIM LPIPS
Baselines: X-NeRF NeSpoF single-modality NeRF
Builds on: NeRF NeRFStudio SDFStudio Neuralangelo
Closest papers by meaning
- NeRF-VPT: Learning Novel View Representations with Neural Radiance Fields via View Prompt Tuning (AAAI 2024 · similarity 0.87)
- GoLF-NRT: Integrating Global Context and Local Geometry for Few-Shot View Synthesis (CVPR 2025 · similarity 0.87)
- Gear-NeRF: Free-Viewpoint Rendering and Tracking with Motion-aware Spatio-Temporal Sampling (CVPR 2024 · similarity 0.86)
- Rad-NeRF: Ray-decoupled Training of Neural Radiance Field (NeurIPS 2024 · similarity 0.86)
- Learning with Unreliability: Fast Few-shot Voxel Radiance Fields with Relative Geometric Consistency (CVPR 2024 · similarity 0.86)
- Is Vanilla MLP in Neural Radiance Field Enough for Few-shot View Synthesis? (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.