Research map / 3D vision / Gaussian splatting
ThermalGaussian: Thermal 3D Gaussian Splatting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ThermalGaussian, the first multimodal 3D Gaussian Splatting method that jointly renders thermal and RGB images using calibrated registration, multimodal training variants, a thermal smoothness loss, and a Gaussian-count-based multimodal regularization, and releases the RGBT-Scenes dataset.
Problem
Thermal 3D scene reconstruction with NeRF is slow to render, and thermal images have low texture and ghosting that break SfM-based pose estimation, while RGB and thermal modalities need joint modeling without one overfitting or degrading the other.
Approach
Calibrate RGB and thermal cameras with a standard chessboard heated for thermal contrast, initialize from multimodal point clouds, and train either two single-modality Gaussian sets or one Gaussian with separate RGB and thermal spherical harmonics under a loss weighted by the ratio of thermal to total Gaussian counts, plus a thermal smoothness term.
Key results
- OMMG improves average thermal PSNR by about 1.3 dB over 3DGS+MI (25.64 vs 24.31 on RGBT-Scenes test views).
- MSMG improves RGB rendering by about 1.1 dB average PSNR over 3DGS (24.27 vs 23.20).
- MSMG with multimodal regularization reduces storage to 9MB+9MB, about 8% of the 3DGS storage (159MB) for the same content, while increasing FPS to 390/420.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multimodal initialization before training 3DGS on thermal images (ablation) | 3DGS thermal reconstruction · Thermal test views (Avg.) | PSNR: 3DGS: ×; 3DGS+MI: 24.31 | Helped |
| Add thermal smoothness loss (ablation) | 3DGS+MI · Thermal ablation (Avg.) | PSNR: 24.31 -> 24.65 | Helped |
| Fine-tune the RGB-trained Gaussian on thermal images (ablation) | 3DGS with multimodal initialization · Thermal ablation (Avg.) | PSNR: 24.31 -> 24.70 | Helped |
| Train separate RGB and thermal Gaussians jointly (ablation) | ThermalGaussian · Thermal test views (Avg.) | PSNR: 24.31 -> 25.38 | Helped |
| Use one multimodal Gaussian instead of separate modality-specific Gaussians (ablation) | ThermalGaussian · Thermal test views (Avg.) | PSNR: 25.38 -> 25.64 | Helped |
| Add multimodal regularization to MSMG (ablation) | ThermalGaussian MSMG · Thermal ablation (Avg.) | PSNR: 25.38 -> 25.09 | Hurt |
| Add multimodal regularization to MSMG (ablation) | ThermalGaussian MSMG · RGB ablation (Avg.) | PSNR: 24.27 -> 24.21 | Hurt |
| Add multimodal regularization to MSMG (ablation) | ThermalGaussian MSMG · Model storage | storage: 18MB+66MB -> 9MB+9MB | Helped |
| Use one multimodal Gaussian rather than separate RGB and thermal Gaussians (ablation) | ThermalGaussian · RGB test views (Avg.) | PSNR: 24.27 -> 24.34 | Mixed |
| Jointly constrain RGB and thermal rendering | 3DGS · RGB test views (Avg.) | PSNR: 23.20 -> 24.34 | Helped |
| Use ThermalGaussian OMMG instead of 3DGS+MI | Thermal rendering · Thermal test views (Avg.) | PSNR: 24.31 -> 25.64 | Helped |
Surprising findings
- Multimodal training with thermal supervision improved RGB rendering quality over RGB-only 3DGS, not just thermal quality.
- A standard RGB chessboard can calibrate thermal and color cameras when heated, avoiding specialized metallic calibration boards.
Limitations
- Thermal images with low texture and ghosting make SfM pose estimation fail, requiring multimodal registration workarounds.
- 3DGS+MI and single-modality variants fail on several scenes where thermal-only localization is impossible.
Experimental setup
Datasets: RGBT-Scenes
Benchmarks: RGBT-Scenes
Metrics: PSNR SSIM LPIPS training time FPS memory
Baselines: 3DGS ThermoNeRF 3DGS+MI
Builds on: 3D Gaussian Splatting NeRF ThermoNeRF Thermal-NeRF
Closest papers by meaning
- 3D-HGS: 3D Half-Gaussian Splatting (CVPR 2025 · similarity 0.88)
- Compact 3D Gaussian Representation for Radiance Field (CVPR 2024 · similarity 0.88)
- FewViewGS: Gaussian Splatting with Few View Matching and Multi-stage Training (NeurIPS 2024 · similarity 0.87)
- 3D Convex Splatting: Radiance Field Rendering with 3D Smooth Convexes (CVPR 2025 · similarity 0.87)
- WildGaussians: 3D Gaussian Splatting In the Wild (NeurIPS 2024 · similarity 0.87)
- CityGaussianV2: Efficient and Geometrically Accurate Reconstruction for Large-Scale Scenes (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.