Research map / 3D vision / Gaussian splatting
Feat2GS: Probing Visual Foundation Models with Gaussian Splatting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Feat2GS probes the geometry and texture awareness of visual foundation models by reading out 3D Gaussian Splatting parameters from frozen VFM features trained only with novel-view photometric loss on unposed multi-view images, and derives stronger NVS baselines via feature concatenation.
Problem
Existing 3D probes of visual foundation models rely on 2.5D depth/normal or sparse two-view correspondence with 3D ground truth, which ignores texture awareness and limits evaluation scale and diversity.
Approach
Regress 3D Gaussian parameters from frozen VFM features via a shallow 2-layer MLP readout, with camera poses from DUSt3R, and evaluate novel view synthesis under Geometry, Texture, and All probing modes that separate geometry and texture awareness without 3D labels.
Key results
- Feat2GS with RADIO features reaches 19.73 PSNR / 0.6513 SSIM on all datasets vs. InstantSplat's 18.87 / 0.6044.
- Feat2GS w/ DUSt3R* (fine-tuned) reaches 0.2928 LPIPS vs. InstantSplat's 0.3039, the only variant beating InstantSplat in LPIPS.
- In Geometry mode RADIO ranks top, while MAE and SAM lead in Texture mode; SD performs worst in most metrics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RADIO features in Geometry mode instead of InstantSplat | Feat2GS · All Datasets (casual NVS) | PSNR: 18.87 vs 19.73 | Helped |
| Use RADIO features in Geometry mode instead of InstantSplat | Feat2GS · All Datasets (casual NVS) | LPIPS: 0.3039 vs 0.3143 | Hurt |
| Concatenate all VFM features instead of using RADIO features (ablation) | Feat2GS · All Datasets (casual NVS) | PSNR: 19.73 vs 19.80 | Helped |
| Concatenate all VFM features instead of using RADIO features (ablation) | Feat2GS · All Datasets (casual NVS) | LPIPS: 0.3143 vs 0.3105 | Helped |
| Fine-tune DUSt3R features during warm start (ablation) | Feat2GS w/ DUSt3R · All Datasets (casual NVS) | LPIPS: 0.3247 vs 0.2928 | Helped |
| Fine-tune DUSt3R features during warm start (ablation) | Feat2GS w/ DUSt3R · All Datasets (casual NVS) | PSNR: 19.66 vs 19.75 | Helped |
| Fine-tune DUSt3R features during warm start (ablation) | Feat2GS w/ DUSt3R · All Datasets (casual NVS) | SSIM: 0.6469 vs 0.6561 | Helped |
| Use masked-image-reconstruction features such as MAE for texture readout | Feat2GS Texture mode · Multiple NVS datasets | LPIPS | Helped |
| Use pointmap-trained features rather than depth-trained features | Feat2GS Geometry mode · Multiple NVS datasets | NVS quality | Helped |
| Use concatenated DINOv2, CLIP, and SAM features rather than RADIO features (ablation) | Feat2GS Geometry mode · Multiple datasets | NVS quality | Helped |
| Concatenate RADIO, MAE, and IUVRGB features instead of using MASt3R features (ablation) | Feat2GS All mode · Multiple datasets | NVS quality | Helped |
| Use VFM features for texture regression in All mode rather than freely optimizing texture in Geometry mode (ablation) | Feat2GS · Multiple NVS datasets | LPIPS: +0.05 | Hurt |
Surprising findings
- Feature concatenation of DINOv2, CLIP and SAM matches RADIO's geometry awareness, and concatenating all VFMs improves further.
- Depth-supervised MiDaS yields much worse geometry than pointmap-trained DUSt3R on the same ViT-L/16 backbone.
Limitations
- Low-resolution VFM features limit LPIPS; feature upsampling gave little benefit.
- Texture mode performance is poor across VFMs, so All mode is blurred.
- Dataset evaluation bias persists; no single VFM wins consistently across datasets.
Experimental setup
Datasets: LLFF DTU DL3DV MipNeRF360 MVImgNet Tanks and Temples Casual
Benchmarks: LLFF DTU DL3DV MipNeRF360 MVImgNet Tanks and Temples Casual
Metrics: PSNR SSIM LPIPS accuracy completeness distance
Baselines: InstantSplat Probe3D
Builds on: 3D Gaussian Splatting DUSt3R InstantSplat Probe3D
Closest papers by meaning
- Probing the 3D Awareness of Visual Foundation Models (CVPR 2024 · similarity 0.91)
- PF3plat: Pose-Free Feed-Forward 3D Gaussian Splatting for Novel View Synthesis (ICML 2025 · similarity 0.88)
- VA-GS: Enhancing the Geometric Representation of Gaussian Splatting via View Alignment (NeurIPS 2025 · similarity 0.88)
- Quantifying and Alleviating Co-Adaptation in Sparse-View 3D Gaussian Splatting (NeurIPS 2025 · similarity 0.87)
- Neural Signed Distance Function Inference through Splatting 3D Gaussians Pulled on Zero-Level Set (NeurIPS 2024 · similarity 0.87)
- GaussianUDF: Inferring Unsigned Distance Functions through 3D Gaussian Splatting (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.