Research map / Image generation / Autoregressive and latent generation
Enhancing High-Resolution 3D Generation through Pixel-wise Gradient Clipping
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Pixel-wise Gradient Clipping (PGC), a plug-in that clips per-pixel gradients flowing through the frozen VAE encoder in latent-diffusion SDS, improving high-resolution texture quality of existing text-to-3D and image-to-3D pipelines (notably enabling SDXL guidance).
Problem
High-resolution 3D generation with SDS through latent diffusion models suffers from uncontrolled pixel-wise gradients propagated through the frozen VAE encoder, which blur or erase texture details.
Approach
Apply norm-based gradient clipping at the level of individual pixel residuals (rather than parameters) on the gradient passing through the VAE, with a threshold near the bounded pixel-residual variance, so texture-bearing gradient directions are preserved while noisy magnitudes are truncated.
Key results
- In a user study with 180 picks, the Fantasia3D+SDXL+PGC method received 84.44% preference versus 10.56% for Fantasia3D with PGC and 5% for Fantasia3D without PGC.
- Directly replacing Stable Diffusion with SDXL in Fantasia3D fails consistently, while adding PGC activates SDXL to yield high-quality textured meshes.
- PGC is reported to benefit Stable-DreamFusion (better detail and success rate) and Fantasia3D geometry stage (fewer mesh fragments), with modest gains for Zero123 at 256 resolution.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add PGC to Fantasia3D with SDXL | Fantasia3D+SDXL · Fixed-mesh texture generation | texture quality | Helped |
| Add PGC to Fantasia3D with Stable Diffusion | Fantasia3D · Fixed-mesh texture generation | texture detail | Helped |
| Add PGC to Stable-DreamFusion | Stable-DreamFusion with Stable Diffusion 2.1-base · Text-to-3D generation | generation details and success rate | Helped |
| Add PGC to Fantasia3D geometry stage | Fantasia3D with Stable Diffusion 2.1-base · 3D geometry generation | optimization stability | Helped |
| Add PGC to Zero123-SDS | Zero123-SDS · Image-to-3D generation | texture quality | Helped |
| Replace Stable Diffusion with SDXL without adding PGC | Fantasia3D · Fixed-mesh texture generation | texture quality | Hurt |
| Use linear VAE approximation instead of the VAE encoder gradients (ablation) | LDM-based SDS · 3D texture generation | texture detail | Hurt |
| Use pixel-wise normalized gradient descent (PNGD) instead of PGC-N (ablation) | SDS-based 3D generation · Texture generation near the clipping threshold | texture detail | Hurt |
| Clip pixel-wise gradients by value instead of by norm (ablation) | Pixel-wise gradient clipping · 3D texture generation | texture quality | Hurt |
| Vary the PGC clipping threshold among 0.01, 0.05, 0.1, 0.5, and 1.0 (ablation) | PGC · 3D texture generation | texture quality | No clear effect |
| Add normal-SDS loss (ablation) | Fantasia3D+SDXL+PGC · Mesh shape generation | shape detail | Helped |
Surprising findings
- Replacing Stable Diffusion with SDXL in Fantasia3D consistently fails without PGC, despite SDXL's stronger high-resolution capability.
- A linear approximation of the VAE (inspired by Latent-NeRF) falls short at capturing fine texture details.
Limitations
- Zero123 gains are modest due to 256 resolution constraint
- Linear VAE approximation fails to capture texture details
- User study is small (15 respondents, 12 meshes)
Experimental setup
Datasets: COCO
Metrics: user preference rate
Baselines: Fantasia3D Fantasia3D+SDXL Stable-DreamFusion Zero123
Builds on: DreamFusion Fantasia3D Magic3D Latent-NeRF Stable Diffusion
Closest papers by meaning
- LT3SD: Latent Trees for 3D Scene Diffusion (CVPR 2025 · similarity 0.89)
- CAD: Photorealistic 3D Generation via Adversarial Distillation (CVPR 2024 · similarity 0.88)
- ScaleCrafter: Tuning-free Higher-Resolution Visual Generation with Diffusion Models (ICLR 2024 · similarity 0.88)
- A 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale (NeurIPS 2026 · similarity 0.88)
- DiffuseHigh: Training-Free Progressive High-Resolution Image Synthesis Through Structure Guidance (AAAI 2025 · similarity 0.88)
- HIFA: High-fidelity Text-to-3D Generation with Advanced Diffusion Guidance (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.