Research map / 3D vision / Gaussian splatting
CLIPGaussian: Universal and Multimodal Style Transfer Based on Gaussian Splatting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLIPGaussian, a plug-in CLIP- and VGG-guided fine-tuning method that stylizes Gaussian Splatting representations of 2D images, videos, 3D objects/scenes and 4D scenes from text or image prompts while keeping the number of Gaussians fixed.
Problem
Style transfer for Gaussian Splatting-based representations is limited mostly to color/opacity edits, and existing methods either handle only one modality, need large diffusion models, or increase model size; a unified text- and image-guided approach across 2D, video, 3D and 4D is lacking.
Approach
Fine-tune the parameters of pretrained GS-based base models (3DGS, D-MiSo, MiraGe, VeGaS) with a composite loss combining CLIP directional loss on whole renders and random patches, a VGG-19 content loss, and a background loss, updating all Gaussian parameters without densification so model size is preserved and style can be interpolated.
Key results
- Under text conditioning, CLIPGaussian reaches CLIP-S 26.86 and CLIP-SIM 26.31 versus 17.59 and 12.27 for the best text baseline DGE, without increasing the number of Gaussians.
- Under image conditioning, CLIPGaussian scores CLIP-S 72.65 and CLIP-SIM 20.72, below G-Style (76.94 and 24.94) but with +0% memory versus +126% for G-Style.
- In a CLICKworker user study with 30 participants per survey, the method was most often chosen as most similar to the reference for object stylization with images and was rated best for text-based object stylization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply CLIPGaussian instead of DGE | 3D Gaussian Splatting style transfer · 3D style transfer, text-conditioned | CLIP-S: 17.59 -> 26.86 | Helped |
| Apply CLIPGaussian instead of DGE | 3D Gaussian Splatting style transfer · 3D style transfer, text-conditioned | CLIP-SIM: 12.27 -> 26.31 | Helped |
| Apply CLIPGaussian instead of G-Style | 3D Gaussian Splatting style transfer · 3D style transfer, image-conditioned | CLIP-S: 76.94 -> 72.65 | Hurt |
| Apply CLIPGaussian instead of G-Style | 3D Gaussian Splatting style transfer · 3D style transfer, image-conditioned | Memory size change: +126% -> +0% | Helped |
| Use lighter stylization hyperparameters instead of standard CLIPGaussian (ablation) | CLIPGaussian · 3D style transfer, text-conditioned | CLIP-S: 26.86 -> 23.14 | Hurt |
| Use lighter stylization hyperparameters instead of standard CLIPGaussian (ablation) | CLIPGaussian · 3D style transfer, text-conditioned | CLIP-F: 98.80 -> 99.17 | Helped |
| Use lighter stylization hyperparameters instead of standard CLIPGaussian (ablation) | CLIPGaussian · 3D style transfer, image-conditioned | CLIP-S: 72.65 -> 69.27 | Hurt |
| Use lighter stylization hyperparameters instead of standard CLIPGaussian (ablation) | CLIPGaussian · 3D style transfer, image-conditioned | CLIP-F: 98.78 -> 99.16 | Helped |
Surprising findings
- The method is slightly below G-Style on image-conditioned CLIP-S and CLIP-SIM, and the paper itself notes its 2D results may not match diffusion-based models in visual quality.
- The text-conditioned CLIPGaussian has the worst CLIP-CONS score (2.34) among text-conditioned methods, a lower consistency value than I-GS2GS (13.53) and DGE (12.46).
Limitations
- 2D image stylization quality is below large generative and diffusion-based methods.
- Performance depends on the quality of the underlying base GS model; poor reconstructions can make style transfer implausible.
- 4D comparison with 4DStyleGaussian relies on qualitative results from the paper because its code was not public.
Experimental setup
Datasets: NeRF-Synthetic Mip-NeRF 360 MS-COCO DAVIS UVG Neural 3D Video (DyNeRF) D-NeRF
Benchmarks: NeRF-Synthetic Mip-NeRF 360 DyNeRF coffee_martini D-NeRF MS-COCO DAVIS UVG
Metrics: CLIP-S CLIP-SIM CLIP-F CLIP-CONS memory size (number of Gaussians) short- and long-range LPIPS/RMSE consistency optical flow consistency user study ranking
Baselines: I-GS2GS DGE StyleGaussian SGSST ABC-GS G-Style 4DStyleGaussian Instruct 4D-to-4D AdaIN StyTr^2 InstructPix2Pix CCPL
Builds on: 3D Gaussian Splatting CLIPstyler VeGaS D-MiSo MiraGe
Closest papers by meaning
- Multi-StyleGS: Stylized Gaussian Splatting with Multiple Styles (AAAI 2025 · similarity 0.92)
- SGSST: Scaling Gaussian Splatting Style Transfer (CVPR 2025 · similarity 0.92)
- Morpheus: Text-Driven 3D Gaussian Splat Shape and Color Stylization (CVPR 2025 · similarity 0.90)
- ReGS: Reference-based Controllable Scene Stylization with Gaussian Splatting (NeurIPS 2024 · similarity 0.90)
- Textured Gaussians for Enhanced 3D Scene Appearance Modeling (CVPR 2025 · similarity 0.88)
- DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.