Research map / Image generation / Editing and style transfer
Single Mesh Diffusion Models with Field Latents for Texture Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Field Latents (tangent-vector latent codes on mesh vertices, via a locally-supported VAE) and Field Latent Diffusion Models built on field convolutions, enabling intrinsic single-textured-mesh texture generation, inpainting, label-guided generation, and isometry-equivariant texture transfer.
Problem
Generating high-fidelity textures directly on 3D mesh surfaces, avoiding view inconsistency and rasterization aliasing of image-based or triplane-based methods, in the single-textured-mesh setting where only one example texture is available.
Approach
Encode textures as isometry-equivariant tangent vector fields (Field Latents) at mesh vertices using a log-map-based neural field decoder, and train a DDPM with field convolutions (with embeddings injected into convolutions to preserve equivariance) directly in this intrinsic latent space.
Key results
- FL-VAE reconstructs textures at 30K vertices with PSNR 22.38 vs 18.86 for INFs (LPIPS 1.02 vs 2.15, scaled).
- On unconditional generation over 10 assets, FLDM achieves SIFID 3.27 vs 6.58 for Sin3DM (lower is better), with LPIPS diversity 1.15 vs 2.20.
- FLDMs support label-guided generation, mask-based inpainting, and generative texture transfer across approximately isometric meshes, including different genus.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the logarithmic coordinate function instead of barycentric interpolation (ablation) | FL-VAE · Google Scanned Objects high-res (30K vertices) | PSNR: 21.33 -> 22.38 | Helped |
| Use the logarithmic coordinate function instead of barycentric interpolation (ablation) | FL-VAE · Google Scanned Objects high-res (30K vertices) | DSSIM: 0.66 -> 0.51 | Helped |
| Use the logarithmic coordinate function instead of barycentric interpolation (ablation) | FL-VAE · Google Scanned Objects high-res (30K vertices) | LPIPS: 1.31 -> 1.02 | Helped |
| Use the logarithmic coordinate function instead of barycentric interpolation (ablation) | FL-VAE · Google Scanned Objects low-res (5K vertices) | PSNR: 19.61 -> 20.59 | Helped |
| Use the logarithmic coordinate function instead of barycentric interpolation (ablation) | FL-VAE · Google Scanned Objects low-res (5K vertices) | DSSIM: 1.01 -> 0.83 | Helped |
| Use the logarithmic coordinate function instead of barycentric interpolation (ablation) | FL-VAE · Google Scanned Objects low-res (5K vertices) | LPIPS: 2.03 -> 1.81 | Helped |
| Use FL-VAE rather than INFs for texture reconstruction | Texture reconstruction · Google Scanned Objects high-res (30K vertices) | PSNR: 18.86 -> 22.38 | Helped |
| Use FL-VAE rather than INFs for texture reconstruction | Texture reconstruction · Google Scanned Objects high-res (30K vertices) | DSSIM: 1.16 -> 0.51 | Helped |
| Use FL-VAE rather than INFs for texture reconstruction | Texture reconstruction · Google Scanned Objects high-res (30K vertices) | LPIPS: 2.15 -> 1.02 | Helped |
| Use FLDM rather than Sin3DM for unconditional texture generation | Unconditional generation · Selected Objaverse and Scanned Objects meshes | SIFID: 6.58 -> 3.27 | Helped |
Surprising findings
- Sin3DM attains higher LPIPS diversity than FLDM, but the paper attributes this to Sin3DM also synthesizing geometry rather than textures alone.
Limitations
- Relatively long training time for FLDMs.
- Cannot reflect directional information in synthesized textures.
- Single-textured-mesh setting limits generalization to broad texture distributions.
Experimental setup
Datasets: OpenImagesV4 Google Scanned Objects Objaverse
Benchmarks: Google Scanned Objects reconstruction Objaverse and Scanned Objects unconditional generation
Metrics: PSNR DSSIM LPIPS SIFID
Baselines: Sin3DM INFs FL-VAE (Barycentric)
Builds on: DDPM Intrinsic Neural Fields field convolutions SinDiffusion VN-Transformer
Closest papers by meaning
- Sin3DM: Learning a Diffusion Model from a Single 3D Textured Shape (ICLR 2024 · similarity 0.91)
- UV-free Texture Generation with Denoising and Geodesic Heat Diffusion (NeurIPS 2024 · similarity 0.89)
- One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion (CVPR 2024 · similarity 0.89)
- GenesisTex: Adapting Image Denoising Diffusion to Texture Space (CVPR 2024 · similarity 0.88)
- TriTex: Learning Texture from a Single Mesh via Triplane Semantic Features (CVPR 2025 · similarity 0.88)
- LT3SD: Latent Trees for 3D Scene Diffusion (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.