Research map / 3D vision / 3D generation
PM-INR: Prior-Rich Multi-Modal Implicit Large-Scale Scene Neural Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PM-INR, a multi-modal prior-rich implicit neural representation that injects image, text (CLIP), and 3D point-cloud codebook priors into Mip-NeRF 360 sampling regions to model outdoor large-scale unbounded scenes.
Problem
Implicit neural representations for block- or city-scale outdoor unbounded scenes degrade because point/region sampling cannot cover the cubically growing sampling space, leaving global context missing.
Approach
Extract codebook priors from VQ-VAE image codes, CLIP text embeddings, and sparse-point-cloud 3D codes via learnable query cross-attention, fuse them with cross-attention into a unified cross-modal prior, and inject that prior into each sampling region of a Mip-NeRF 360 backbone so regions gain global semantic information without densifying samples.
Key results
- On OMMO, PM-INR reaches 27.10 PSNR / 0.81 SSIM / 0.239 LPIPS versus 23.10 / 0.67 / 0.419 for Mip-NeRF 360, the strongest baseline.
- On five BlendedMVS large-scale scenes, PM-INR also reports 27.10 PSNR / 0.81 SSIM / 0.239 LPIPS against Mip-NeRF 360's 23.10 / 0.67 / 0.419.
- Ablation: the full multi-modal model reaches 31.34 PSNR, versus 31.12 without cross-attention fusion and 28.34 for direct addition fusion.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add multi-modal priors to the Mip-NeRF 360 backbone | Mip-NeRF 360 · OMMO | PSNR: 23.10 -> 27.10 | Helped |
| Add multi-modal priors to the Mip-NeRF 360 backbone | Mip-NeRF 360 · OMMO | SSIM: 0.67 -> 0.81 | Helped |
| Add multi-modal priors to the Mip-NeRF 360 backbone | Mip-NeRF 360 · OMMO | LPIPS: 0.419 -> 0.239 | Helped |
| Add multi-modal priors to the Mip-NeRF 360 backbone | Mip-NeRF 360 · BlendedMVS (five outdoor unbounded large-scale scenes) | PSNR: 23.10 -> 27.10 | Helped |
| Add multi-modal priors to the Mip-NeRF 360 backbone | Mip-NeRF 360 · BlendedMVS (five outdoor unbounded large-scale scenes) | SSIM: 0.67 -> 0.81 | Helped |
| Add multi-modal priors to the Mip-NeRF 360 backbone | Mip-NeRF 360 · BlendedMVS (five outdoor unbounded large-scale scenes) | LPIPS: 0.419 -> 0.239 | Helped |
| Remove cross-attention from multi-modal prior fusion (ablation) | PM-INR · OMMO ablation | PSNR: 31.12 -> 31.338 | Hurt |
| Inject modalities serially rather than using the proposed fusion strategy (ablation) | PM-INR · OMMO ablation | PSNR: 29.36 -> 31.338 | Hurt |
| Directly add the modality priors (ablation) | PM-INR · OMMO ablation | PSNR: 28.34 -> 31.338 | Hurt |
| Concatenate modality priors along the feature dimension instead of the zeroth dimension (ablation) | PM-INR · OMMO ablation | PSNR: 28.64 -> 31.338 | Hurt |
| Remove all modal priors (ablation) | PM-INR · OMMO | PSNR: 27.75 -> 31.34 | Hurt |
| Use image prior only instead of the multi-modal priors (ablation) | PM-INR · OMMO | PSNR: 28.03 -> 31.34 | Hurt |
Surprising findings
- The paper's BlendedMVS table reports numbers identical to the OMMO table for both Mip-NeRF 360 and PM-INR, which raises a consistency question about the reported results.
Limitations
- No scene editing capability.
- Per-scene optimization limits the number of scenes evaluated (BlendedMVS restricted to five scenes).
Experimental setup
Datasets: OMMO BlendedMVS
Benchmarks: OMMO BlendedMVS
Metrics: PSNR SSIM LPIPS
Baselines: NeRF NeRF++ Mip-NeRF Mip-NeRF 360 Mega-NeRF Ref-NeRF
Builds on: Mip-NeRF 360 CoCo-INR VQ-VAE CLIP MinkowskiEngine
Closest papers by meaning
- LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias (ICLR 2025 · similarity 0.87)
- Novel View Synthesis with Pixel-Space Diffusion Models (CVPR 2025 · similarity 0.85)
- Decompositional Neural Scene Reconstruction with Generative Diffusion Prior (CVPR 2025 · similarity 0.85)
- 3D Reconstruction with Generalizable Neural Fields using Scene Priors (ICLR 2024 · similarity 0.85)
- MMAR: Towards Lossless Multi-Modal Auto-Regressive Probabilistic Modeling (CVPR 2025 · similarity 0.85)
- UMAMI: Unifying Masked Autoregressive Models and Deterministic Rendering for View Synthesis (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.