Research map / 3D vision / 3D generation
X-Ray: A Sequential 3D Representation For Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces X-Ray, a multi-layer ray-cast surface representation packed as a video-like tensor, and a two-stage video-diffusion pipeline (X-Ray diffusion model plus upsampler, then Poisson meshing) that generates complete visible and hidden surfaces from a single image.
Problem
Rendering-based 3D generators supervised by 2D renderings only reconstruct visible outer surfaces, leaving object interiors and hidden surfaces incomplete; existing 3D representations trade off generality, accuracy and efficiency.
Approach
Encode a mesh as sequential ray-cast surface hits (hit, depth, normal, color per layer) so that the object becomes a multi-frame video tensor, then reuse video diffusion (Stable Video Diffusion 3D U-Net) and a video upsampler to generate it from one image, decoding back to a mesh via point cloud and Screened Poisson reconstruction.
Key results
- On GSO, Chamfer distance drops from 0.084 (TripoSR) to 0.056 and FS@0.1 rises from 0.878 to 0.973.
- On ShapeNet Car generation, 1-NNA (CD) of 50.69% and Coverage (CD) of 55.82% versus DiT-3D's 51.04% and 50.00%.
- Generation of one 3D object takes about 7 seconds at inference (1 s diffusion, 1 s upsampler, 5 s mesh decoding).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use X-Ray instead of TripoSR | Image-to-3D reconstruction · GSO | CD: 0.084 -> 0.056 | Helped |
| use X-Ray instead of TripoSR | Image-to-3D reconstruction · GSO | FS@0.1: 0.878 -> 0.973 | Helped |
| use X-Ray instead of TripoSR | Image-to-3D reconstruction · OmniObject3D | CD: 0.080 -> 0.054 | Helped |
| use X-Ray instead of TripoSR | Image-to-3D reconstruction · OmniObject3D | FS@0.1: 0.892 -> 0.972 | Helped |
| use X-Ray instead of DiT-3D | 3D generation · ShapeNet Car (CD) | 1-NNA: 51.04 -> 50.69 | Helped |
| use X-Ray instead of DiT-3D | 3D generation · ShapeNet Car (EMD) | 1-NNA: 51.65 -> 50.83 | Helped |
| use X-Ray instead of DiT-3D | 3D generation · ShapeNet Car (CD) | COV: 50.00 -> 55.82 | Helped |
| use X-Ray instead of DiT-3D | 3D generation · ShapeNet Car (EMD) | COV: 56.38 -> 60.27 | Helped |
| reduce the UNet to 10% of original parameters and randomly initialize it (ablation) | X-Ray diffusion model · GSO | CD: 0.099 -> 0.056 | Helped |
| reduce the UNet to 10% of original parameters and randomly initialize it (ablation) | X-Ray diffusion model · GSO | FS@0.1: 0.806 -> 0.973 | Helped |
| finetune the original UNet instead of randomly initializing it (ablation) | X-Ray diffusion model · GSO | CD: 0.099 -> 0.095 | No clear effect |
| add the Hit attribute (ablation) | X-Ray diffusion model · GSO | CD: 0.074 -> 0.068 | Helped |
Surprising findings
- Fine-tuning the pretrained SVD UNet did not beat a randomly initialized UNet with 10% of the parameters, which the authors attribute to the domain gap between video and X-Ray data.
- Stated 96.88% data reduction versus a 256^3 dense voxel grid.
Limitations
- Fixed maximum of 8 layers causes surfaces behind layer 8 to be omitted, so complex objects (e.g. hamburger) lose parts.
- Generated X-Rays have sparse posterior layers that are redundant, and meshes lack smoothness and can be missing parts when truncated.
Experimental setup
Datasets: Objaverse Google Scanned Objects OmniObject3D ShapeNet Car
Benchmarks: Google Scanned Objects OmniObject3D ShapeNet Car
Metrics: Chamfer Distance F-Score@0.1 1-NNA Coverage Earth Mover's Distance
Baselines: One-2-3-45 ZeroShape TGS OpenLRM TripoSR PointFlow DPM PVD LION GET3D MeshDiffusion DiT-3D
Builds on: Stable Video Diffusion diffusion models Screened Poisson reconstruction Objaverse
Closest papers by meaning
- 3DTopia-XL: Scaling High-quality 3D Asset Generation via Primitive Diffusion (CVPR 2025 · similarity 0.85)
- CAT3D: Create Anything in 3D with Multi-View Diffusion Models (NeurIPS 2024 · similarity 0.85)
- One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion (CVPR 2024 · similarity 0.85)
- World Tracing: Pixel-Aligned Geometry Beyond the Visible (NeurIPS 2026 · similarity 0.85)
- Object-X: Learning to Reconstruct Multi-Modal 3D Object Representations (NeurIPS 2025 · similarity 0.85)
- Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D Prior (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.