Research map / 3D vision / Gaussian splatting

Cross-Spectral Gaussian Splatting with Spatial Occupancy Consistency

AAAI 2025 · 3D vision › Gaussian splatting · approach: Sparse-view Gaussian splatting for novel views · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SOC-GS, a 3D Gaussian Splatting pipeline for cross-spectral (RGB, multispectral, infrared) scene representation that shares Gaussian geometry across spectra, initializes and jointly optimizes per-view poses, and uses a cross-spectral densify controller.

Problem

Training a unified cross-spectral scene representation from asynchronously captured images from cameras with different spectral sensitivities is hard: fixed-rig assumptions break with asynchronous capture, and prior NeRF-based X-NeRF is slow and does not scale to large scenes.

Approach

Share explicit Gaussian geometry (mean, covariance, opacity) across spectra with per-spectrum spherical harmonic colors, estimate each view's extrinsics independently via LoFTR matching with ellipsoid-substantiated Gaussians plus PnP, then jointly refine poses and Gaussians, and extend the adaptive densify controller so out-of-FOV regions are filled per spectrum.

Key results

• On the X-NeRF dataset, SOC-GS (tri-modality set) averages a PSNR/SSIM of 34.8/0.92 on RGB, outperforming X-NeRF's 31.7/0.89 on RGB and improving the cross-spectral pose EM.
• On RealSense, SOC-GS reaches 25.1 dB... (reported as average SSIM improvements over X-NeRF) and about 120 FPS at 1 MP cross-spectral rendering versus 0.16 FPS for X-NeRF.
• Table 3 reports SOC-GS at 119 FPS for IR 1.0M vs 0.16 FPS for X-NeRF, roughly two to three orders of magnitude faster.

Measured effects reported in the paper

Surprising findings

• Removing the joint optimization stage improves MS rendering but hurts infrared, indicating the two-stage pipeline trades off spectra differently.
• Preserving pre-trained Gaussians instead of splitting them per spectrum improves RGB and MS but reduces infrared quality.

Limitations

• Pose-initialization and joint optimization take substantial time per view (about 2 s per view for initialization), and the pipeline depends on LoFTR matches that may be unstable across spectra.
• Gaussians are assumed to approximate the same surface across spectra; the authors note geometry from cross-spectral views is not yet optimized.
• Evaluation covers only a small number of scenes (16 X-NeRF scenes and 5 RealSense scenes) and a single RTX3090.

Experimental setup

Datasets: X-NeRF dataset RealSense dataset (self-collected)

Benchmarks: X-NeRF dataset RealSense dataset

Metrics: PSNR SSIM Euclidean Metric (EM) FPS mutual information (MI)

Baselines: X-NeRF NeRF 3DGS Nope-NeRF Flowmap

Builds on: 3D Gaussian Splatting X-NeRF NeRF-- iComMa

Closest papers by meaning

• COLMAP-Free 3D Gaussian Splatting (CVPR 2024 · similarity 0.86)
• Structure Consistent Gaussian Splatting with Matching Prior for Few-shot Novel View Synthesis (NeurIPS 2024 · similarity 0.86)
• IBGS: Image-Based Gaussian Splatting (NeurIPS 2025 · similarity 0.85)
• FewViewGS: Gaussian Splatting with Few View Matching and Multi-stage Training (NeurIPS 2024 · similarity 0.85)
• Holistic Large-Scale Scene Reconstruction via Mixed Gaussian Splatting (NeurIPS 2025 · similarity 0.85)
• VA-GS: Enhancing the Geometric Representation of Gaussian Splatting via View Alignment (NeurIPS 2025 · similarity 0.85)