Research map / Point clouds / Point cloud classification
UniPre3D: Unified Pre-training of 3D Point Cloud Models with Cross-Modal Gaussian Splatting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UniPre3D, a unified generative pre-training method that predicts Gaussian primitives from point clouds, renders views with differentiable 3D Gaussian splatting, and fuses Stable Diffusion autoencoder image features, applicable to both object- and scene-level point clouds.
Problem
Point cloud pre-training is split by data scale: MAE-based methods work for object-level clouds but fail on scenes, while contrastive methods work for scenes but saturate on objects. No unified 3D pre-training method is effective across both scales and architectures.
Approach
Use 3D Gaussian splatting as a differentiable renderer so the 3D backbone predicts Gaussian primitives and is supervised by pixel-wise MSE on projected images; adapt task difficulty with scale-adaptive fusion of pre-trained 2D image features (feature fusion for objects, point fusion with back-projected depth pixels for scenes).
Key results
- On ScanObjectNN PB_T50_RS, UniPre3D lifts the standard Transformer from 77.24 to 87.93 overall accuracy, above TAP (85.67) and Point-CMAE (85.95).
- With PointTransformerV3 on ScanNet200, UniPre3D improves semantic segmentation mIoU from 35.2 to 36.0.
- With SparseUNet on ScanNetV2 semantic segmentation, UniPre3D reaches 75.8 mIoU, and 33.0 on ScanNet200, both above prior contrastive methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pre-train with UniPre3D instead of no pre-training | Standard Transformer · ScanObjectNN PB T50 RS | accuracy: 77.24 -> 87.93 | Helped |
| pre-train PointMLP with UniPre3D instead of TAP | PointMLP · ScanObjectNN PB T50 RS | accuracy: 88.5 -> 89.5 | Helped |
| pre-train Mamba3D with UniPre3D | Mamba3D · ScanObjectNN PB T50 RS | accuracy: 92.6 -> 93.4 | Helped |
| pre-train PointMLP with UniPre3D instead of TAP | PointMLP · ShapeNetPart | mIoUI: 86.9 -> 86.8 | Hurt |
| pre-train PTv3 with UniPre3D | PTv3 · ScanNet20 | mIoU: 77.5 -> 77.6 | Helped |
| pre-train PTv3 with UniPre3D | PTv3 · ScanNet200 | mIoU: 35.2 -> 36.0 | Helped |
| add image-feature fusion at Decoder-Last (ablation) | Standard Transformer · ScanObjectNN PB T50 RS | accuracy: 86.6 -> 87.9 | Helped |
| add image-feature fusion at Decoder-Mid (ablation) | Standard Transformer · ScanObjectNN PB T50 RS | accuracy: 86.6 -> 87.0 | Helped |
| add image-feature fusion at all decoder layers (ablation) | Standard Transformer · ScanObjectNN PB T50 RS | accuracy: 86.6 -> 86.5 | Hurt |
| add image-feature fusion at Decoder-Last (ablation) | PointMLP · ScanObjectNN PB T50 RS | accuracy: 89.2 -> 89.5 | Helped |
| add image-feature fusion at all decoder layers (ablation) | PointMLP · ScanObjectNN PB T50 RS | accuracy: 89.2 -> 88.9 | Hurt |
| replace feature fusion with point fusion at encoder (ablation) | SparseUNet · ScanNet200 | mIoU: 32.3 -> 33.0 | Helped |
Surprising findings
- Adding excessive image-model information can raise pre-training PSNR yet hurt downstream fine-tuning, per the integration-layer ablation.
- Feature fusion from 2D images did not help scene-level pre-training; point fusion at the encoder layer was needed.
Limitations
- Does not address scenarios beyond object and scene scales.
- Manual selection of fusion strategy (feature vs point fusion) limits unification.
- Requires both point clouds and images, adding data curation burden.
Experimental setup
Datasets: ShapeNet ScanNetV2 ScanObjectNN ShapeNetPart ScanNet200 S3DIS
Benchmarks: ScanObjectNN ShapeNetPart ScanNet20 ScanNet200 S3DIS
Metrics: overall accuracy mIoU mAP PSNR
Baselines: Point-MAE Point-CMAE PointDif TAP PonderV2 PPT PointContrast CSC MSC
Builds on: 3D Gaussian Splatting Point-MAE PonderV2 TAP Point Transformer V3
Closest papers by meaning
- ComPC: Completing a 3D Point Cloud with 2D Diffusion Priors (ICLR 2025 · similarity 0.90)
- Sparse Point Cloud Patches Rendering via Splitting 2D Gaussians (CVPR 2025 · similarity 0.89)
- COLMAP-Free 3D Gaussian Splatting (CVPR 2024 · similarity 0.89)
- GaussianPainter: Painting Point Cloud into 3D Gaussians with Normal Guidance (AAAI 2025 · similarity 0.89)
- Lightweight Predictive 3D Gaussian Splats (ICLR 2025 · similarity 0.89)
- Mitigating Ambiguities in 3D Classification with Gaussian Splatting (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.