Research map / 3D vision / Face and shape
Mosaic-SDF for 3D Generative Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Mosaic-SDF, a set-of-local-grids approximation of signed distance functions, and trains a permutation-equivariant transformer flow-matching model on it for class-conditional and text-to-3D generation.
Problem
Forward-based 3D generative models need a shape representation that is fast to preprocess for large datasets, parameter-efficient, and tensorially simple enough for transformers; existing representations (voxel grids, triplanes, point clouds, meshes, INRs) fail one or more of these.
Approach
Approximate a shape's SDF as a weighted sum of small 7x7x7 local grids centered near the surface (placed by farthest point sampling with a minimal covering scale), giving an n x d matrix per shape that is a set and can be fed to a permutation-equivariant transformer trained with flow matching.
Key results
- On ShapeNetCore-V2 class-conditional generation, M-SDF achieves best or second-best on all metrics across airplane, car, chair, sofa, and table; e.g., airplane FPD 0.44 vs 0.81 for the next best (NW).
- M-SDF fits a 1024x73-style representation (355K parameters) in under 2 minutes per shape on one A100, and gives higher surface approximation quality than INR at the same parameter budget while being far faster to compute.
- Surface extraction at 512^3 takes 21.48 s for M-SDF versus 159.56 s for 3DILG; S2VS runs out of memory at 128^3 and above.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use M-SDF Flow Matching instead of the reported baselines | Class-conditioned ShapeNetCore-V2 generation · airplane | FPD: 0.81 -> 0.44 | Helped |
| Use M-SDF Flow Matching instead of the reported baselines | Class-conditioned ShapeNetCore-V2 generation · car | FPD: 1.32 -> 0.46 | Helped |
| Use M-SDF Flow Matching instead of the reported baselines | Class-conditioned ShapeNetCore-V2 generation · chair | FPD: 0.77 -> 0.52 | Helped |
| Use M-SDF Flow Matching instead of the reported baselines | Class-conditioned ShapeNetCore-V2 generation · sofa | FPD: 1.17 -> 0.63 | Helped |
| Use M-SDF Flow Matching instead of the reported baselines | Class-conditioned ShapeNetCore-V2 generation · table | FPD: 0.83 -> 0.47 | Helped |
| Use M-SDF Flow Matching instead of the reported baselines | Class-conditioned ShapeNetCore-V2 generation · airplane | COV-CD: 51.98 -> 52.48 | Helped |
| Use M-SDF Flow Matching instead of the reported baselines | Class-conditioned ShapeNetCore-V2 generation · car | COV-CD: 37.71 -> 45.71 | Helped |
| Use M-SDF Flow Matching instead of the reported baselines | Class-conditioned ShapeNetCore-V2 generation · chair | COV-CD: 51.78 -> 48.22 | Hurt |
| Use M-SDF Flow Matching instead of the reported baselines | Class-conditioned ShapeNetCore-V2 generation · sofa | COV-CD: 48.73 -> 46.20 | Hurt |
| Use M-SDF Flow Matching instead of the reported baselines | Class-conditioned ShapeNetCore-V2 generation · table | COV-CD: 53.44 -> 52.97 | Hurt |
| Use M-SDF Flow Matching instead of the reported baselines | Class-conditioned ShapeNetCore-V2 generation · airplane | MMD-CD: 3.36 -> 3.54 | Hurt |
| Use M-SDF Flow Matching instead of the reported baselines | Class-conditioned ShapeNetCore-V2 generation · car | MMD-CD: 4.13 -> 2.87 | Helped |
Surprising findings
- Triplane representations degraded in approximation quality as parameter count increased under the shared supervision used here, which the authors attribute to missing regularization used in prior triplane works.
Limitations
- Only encodes SDF geometry; no texture, color, or lighting.
- Uses a simple linear layer to embed local grids; no convolutional or autoencoder compression.
- Permutation-equivariant only; not orientation-equivariant.
Experimental setup
Datasets: ShapeNetCore-V2 600K caption-shape pairs (Cap3D)
Benchmarks: ShapeNetCore-V2 class-conditional generation (5 largest classes)
Metrics: FPD KPD COV MMD 1-NNA Chamfer Distance EMD surface extraction time Chamfer distance for representation quality
Baselines: 3DILG Neural Wavelet 3DShape2VecSet 3D Volumetric Grid Triplane INR
Builds on: Flow Matching 3DShape2VecSet Implicit Neural Representations Classifier-Free Guidance
Closest papers by meaning
- Make-A-Shape: a Ten-Million-scale 3D Shape Model (ICML 2024 · similarity 0.89)
- GPLD3D: Latent Diffusion of 3D Shape Generative Models by Enforcing Geometric and Physical Priors (CVPR 2024 · similarity 0.87)
- Sin3DM: Learning a Diffusion Model from a Single 3D Textured Shape (ICLR 2024 · similarity 0.87)
- Diffusion 3D Features (Diff3F): Decorating Untextured Shapes with Distilled Semantic Features (CVPR 2024 · similarity 0.87)
- ShapeShifter: 3D Variations Using Multiscale and Sparse Point-Voxel Diffusion (CVPR 2025 · similarity 0.87)
- Flow-Corrected Shape Optimization: Taming Manifold Drift in High-Dimensional 3D Models (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.