Research map / 3D vision / 3D generation
Object-X: Learning to Reconstruct Multi-Modal 3D Object Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Object-X learns compact multi-modal object embeddings from voxelized multi-view DINOv2 features that can be decoded into 3D Gaussian Splats and reused for localization and scene alignment.
Problem
Existing object embeddings for 3D scenes are task-specific and cannot be decoded into explicit geometry and appearance, forcing storage of raw images or point clouds alongside them.
Approach
Voxelize an object into a 3D grid with fused DINOv2 features (SLat), compress it into a single dense U-3DGS embedding, and decode it into 3D Gaussian splat parameters, while adding auxiliary modality encoders trained for localization with frozen reconstruction core.
Key results
- On 3RScan object reconstruction, Object-X reaches F1@0.05 of 77.80 versus 44.35 for the best 12-view baseline (3DGS 12V), with 0.14 MB storage versus 6.71 MB and runtime 0.051 s versus 58.5 s.
- On ScanNet (zero-shot, not trained on it), Object-X gets SSIM 0.966 and F1@0.05 of 89.09, runtime 0.033 s.
- On 3RScan scene alignment without task-specific training, Object-X achieves Mean RR 0.910 and Hits@1 0.864, second to SGAligner with object attributes (0.950 / 0.923).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Object-X instead of 3DGS (12V) for object reconstruction | 3RScan object reconstruction · 3RScan | Accuracy@0.05: 33.72 -> 80.22 | Helped |
| Use Object-X instead of 3DGS (12V) for object reconstruction | 3RScan object reconstruction · 3RScan | LPIPS: 0.072 -> 0.065 | Helped |
| Use Object-X instead of 3DGS (12V) for object reconstruction | 3RScan object reconstruction · 3RScan | Completion@0.05: 77.81 -> 77.80 | No clear effect |
| Use Object-X instead of 3DGS (12V) for object reconstruction | 3RScan object reconstruction · 3RScan | PSNR: 31.613 -> 30.981 | Hurt |
| Use Object-X instead of 3DGS (4V) for object reconstruction | ScanNet object reconstruction · ScanNet | Accuracy@0.05: 36.95 -> 88.66 | Helped |
| Use Object-X instead of 3DGS (4V) for object reconstruction | ScanNet object reconstruction · ScanNet | SSIM: 0.949 -> 0.966 | Hurt |
| Use RGB-D input rather than RGB input for Object-X single-image reconstruction | Object-X · 3RScan single-image reconstruction | F1@0.05: 48.397 -> 63.223 | Helped |
| Use Object-X (RGB-D) instead of MIDI | 3RScan single-image reconstruction · 3RScan | F1@0.05: 34.516 -> 63.223 | Helped |
| Add 3DGS optimization after Object-X scene initialization | Object-X scene composition · 3RScan scene composition | F1@0.05: 49.99 -> 66.49 | Helped |
| Add 3DGS optimization after Object-X scene initialization | Object-X scene composition · 3RScan scene composition | SSIM: 0.677 -> 0.727 | Helped |
| Use Object-X without scene-alignment-specific training rather than SGAligner with point-cloud and object modalities | 3RScan scene alignment · 3RScan | Mean RR: 0.950 -> 0.910 | Hurt |
Surprising findings
- Object-X obtains much higher geometric F1 than optimized 3DGS (77.80 vs 41.41 on 3RScan) despite lower PSNR (30.98 vs 34.01).
Limitations
- High compression loses fine details for large or complex objects.
- Zero-shot single-image reconstruction does not consistently match optimized task-specific methods.
- Full-scene composition shows artifacts from missing object segmentations and lower SSIM/PSNR than 3DGS (12V).
Experimental setup
Datasets: 3RScan ScanNet SceneGraphFusion-generated annotations
Benchmarks: 3RScan ScanNet SceneGraphLoc localization protocol on 3RScan SGAligner scene alignment protocol on 3RScan
Metrics: SSIM PSNR LPIPS Accuracy@0.05 Completion@0.05 F1@0.05 runtime storage Recall@1/3/5 Mean RR
Baselines: 3DGS 2DGS DepthSplat MIDI-3D SGAligner EVA CrossOver SceneGraphLoc
Builds on: 3D Gaussian Splatting structured latent (SLat) DINOv2 SceneGraphLoc SGAligner
Closest papers by meaning
- CrossOver: 3D Scene Cross-Modal Alignment (CVPR 2025 · similarity 0.87)
- Cross-Modal Distillation for 2D/3D Multi-Object Discovery from 2D Motion (CVPR 2025 · similarity 0.87)
- Learning Partonomic 3D Reconstruction from Image Collections (CVPR 2025 · similarity 0.87)
- DynaVol: Unsupervised Learning for Dynamic Scenes through Object-Centric Voxelization (ICLR 2024 · similarity 0.87)
- Slice3D: Multi-Slice Occlusion-Revealing Single View 3D Reconstruction (CVPR 2024 · similarity 0.87)
- Cross-Modal 3D Representation with Multi-View Images and Point Clouds (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.