Research map / 3D vision / Face and shape

RaySt3R: Predicting Novel Depth Maps for Zero-Shot Object Completion

NeurIPS 2025 · 3D vision › Face and shape · approach: Single-view 3D reconstruction · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Recasts zero-shot 3D object completion from a single masked RGB-D image as novel-view depth, mask, and confidence prediction by a feedforward transformer, then fuses multi-view predictions into a complete shape.

Problem

Multi-object 3D shape completion from a single RGB-D image in cluttered real-world scenes, where existing methods lack 3D consistency, are computationally expensive, and miss sharp boundaries.

Approach

Cast shape completion as novel view synthesis: a transformer takes the input point map and a query ray map, cross-attends to frozen DINOv2 features, and predicts confidence-aware depth and object masks for many query rays; predictions are occlusion-filtered, mask-filtered, confidence-filtered and fused into a point cloud.

Key results

• Achieves CD 5.21 mm and F1 0.893 on the OctMAE synthetic test split, versus 6.48 mm for the strongest baseline OctMAE.
• Achieves CD 3.56 mm and F1 0.930 on YCB-Video, versus 6.40 mm for OctMAE, with the lowest std across real datasets (1.74 mm).
• Runs in under 1.2 s per scene on a single RTX 4090 and is trained only on synthetic data, reporting up to 44% CD improvement over baselines.

Measured effects reported in the paper

Surprising findings

• Feeding masked RGB images to TRELLIS did not always help; raw inputs outperformed masked on HomebrewedDB.
• ViewCrafter's video-diffusion novel views were too poor and not 3D-consistent to evaluate.

Limitations

• Trained only on synthetic data; authors suggest real-world training could improve results.
• Performance degrades substantially with false-negative input mask errors.
• Limited by synthetic dataset diversity for complex objects.

Experimental setup

Datasets: OctMAE synthetic scenes GSO Objaverse YCB-Video HOPE HomebrewedDB

Benchmarks: OctMAE test split YCB-Video HOPE HomebrewedDB

Metrics: chamfer distance F1-Score@10mm

Baselines: OctMAE VoxFormer ShapeFormer MCC ConvONet POCO AICNet Minkowski OCNN SceneComplete TRELLIS Unique3D

Builds on: DUSt3R DINOv2 OctMAE DPT

Closest papers by meaning

• RayDF: Neural Ray-surface Distance Fields with Multi-view Consistency (NeurIPS 2023 · similarity 0.88)
• Shape Completion Using 3D-Encoder-Predictor CNNs and Shape Synthesis (CVPR 2017 · similarity 0.87)
• DeCoTR: Enhancing Depth Completion with 2D and 3D Attentions (CVPR 2024 · similarity 0.86)
• Zero-Shot Novel View and Depth Synthesis with Multi-View Geometric Diffusion (CVPR 2025 · similarity 0.86)
• Zero-shot Depth Completion via Test-time Alignment with Affine-invariant Depth Prior (AAAI 2025 · similarity 0.86)
• GenPC: Zero-shot Point Cloud Completion via 3D Generative Priors (CVPR 2025 · similarity 0.86)