Research map / 3D vision / 3D generation
MV-CoLight: Efficient Object Compositing with Consistent Lighting and Shadow Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MV-CoLight, a two-stage feed-forward Swin-Transformer framework that harmonizes and relights inserted objects in 2D images and multi-view 3D Gaussian scenes, plus a 480k-scene synthetic dataset (DTC-MultiLight).
Problem
Illumination-consistent object compositing (color harmonization, relighting, shadows) across multiple views of 3D scenes, where existing single-image diffusion methods lack multi-view consistency and inverse-rendering Gaussian methods are slow and rely on imperfect decomposition.
Approach
A 2D feed-forward Swin-Transformer compositing network whose multi-view features are mapped into 3D Gaussian color space via Hilbert-curve ordering, then a second 3D Swin network predicts harmonized Gaussian colors with frozen geometry, avoiding diffusion sampling and per-scene environment-map estimation.
Key results
- Single-view on simple synthetic scenes: Ours reaches 29.65 PSNR / 0.961 SSIM / 0.029 LPIPS vs. best baseline PCT-Net 22.58 PSNR, at 0.07s inference.
- Multi-view on simple synthetic scenes: Ours achieves 30.29 PSNR / 0.960 SSIM, outperforming GI-GS (18.97 PSNR) and IRGS (17.79 PSNR), with about 1.08s inference after Gaussian training.
- Ablation: removing the 2D compositing model drops multi-view PSNR from 30.29 to 25.83 on simple synthetic scenes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MV-CoLight instead of PCT-Net | Single-view object compositing · Simple Synthetic Scene | PSNR: 22.58 -> 29.65 | Helped |
| Use MV-CoLight instead of PCT-Net | Single-view object compositing · Complex Synthetic Scene | PSNR: 25.26 -> 30.20 | Helped |
| Use MV-CoLight instead of PCT-Net | Single-view object compositing · Objects With Lighting | PSNR: 25.08 -> 28.75 | Helped |
| Use MV-CoLight instead of PCT-Net | Multi-view object compositing · Simple Synthetic Scene | PSNR: 22.97 -> 30.29 | Helped |
| Use MV-CoLight instead of PCT-Net | Multi-view object compositing · Complex Synthetic Scene | PSNR: 25.19 -> 30.13 | Helped |
| Use MV-CoLight instead of PCT-Net | Multi-view object compositing · Real Captured Scene | PSNR: 23.39 -> 26.39 | Helped |
| Use 3 transformer blocks instead of 2 (ablation) | MV-CoLight 2D and 3D models · Simple Synthetic Scene | PSNR: 28.34 -> 29.65 (2D); 28.70 -> 30.29 (3D) | Helped |
| Use 6 transformer layers instead of 4 (ablation) | MV-CoLight 2D and 3D models · Simple Synthetic Scene | PSNR: 28.39 -> 29.65 (2D); 28.83 -> 30.29 (3D) | Helped |
| Increase embedding dimension from 60 to the full model setting (ablation) | MV-CoLight 2D and 3D models · Simple Synthetic Scene | PSNR: 27.68 -> 29.65 (2D); 28.11 -> 30.29 (3D) | Helped |
| Add Hilbert transform (ablation) | MV-CoLight 3D model · Simple Synthetic Scene | PSNR: 28.99 -> 30.29 | Helped |
| Add the 2D object compositing model and its features (ablation) | MV-CoLight 3D model · Simple Synthetic Scene | PSNR: 25.83 -> 30.29 | Helped |
| Add depth input (ablation) | MV-CoLight 2D and 3D models · Simple Synthetic Scene | PSNR: 29.11 -> 29.65 (2D); 29.73 -> 30.29 (3D) | Helped |
Surprising findings
- A variant without background or depth input still performs close to the full model (e.g., 29.11 vs 29.65 PSNR single-view), suggesting the model relies mostly on the composite image itself.
- Diffusion-based RGB↔X scores far below feed-forward PCT-Net on single-view metrics (12.28 vs 22.58 PSNR).
Limitations
- Color bias when applied to some real-world scenes due to predominantly synthetic training data.
- Physically inconsistent illumination: specular highlight positions and shadow directions can deviate under complex lighting, since no strict physical constraints are used.
- Gaussian parameterization errors can propagate and degrade harmonization quality.
Experimental setup
Datasets: DTC-MultiLight (synthetic, Blender) FOSCom Objects With Lighting (OWL) Digital Twin Catalog Poly Haven HDR environment maps
Benchmarks: FOSCom Objects With Lighting DTC-MultiLight simple synthetic DTC-MultiLight complex synthetic real captured scenes
Metrics: PSNR SSIM LPIPS inference time memory
Baselines: PCT-Net ObjectStitch ControlCom RGB↔X IC-Light LumiNet GPT-4o GS-IR GI-GS IRGS
Builds on: 3D Gaussian splatting Swin Transformer Hilbert curve ObjectStitch PCT-Net
Closest papers by meaning
- ReCap: Better Gaussian Relighting with Cross-Environment Captures (CVPR 2025 · similarity 0.85)
- MetaGS: A Meta-Learned Gaussian-Phong Model for Out-of-Distribution 3D Scene Relighting (NeurIPS 2025 · similarity 0.85)
- GI-GS: Global Illumination Decomposition on Gaussian Splatting for Inverse Rendering (ICLR 2025 · similarity 0.85)
- Generative Multiview Relighting for 3D Reconstruction under Extreme Illumination Variation (CVPR 2025 · similarity 0.85)
- IllumiNeRF: 3D Relighting Without Inverse Rendering (NeurIPS 2024 · similarity 0.85)
- Eclipse: Disambiguating Illumination and Materials using Unintended Shadows (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.