Research map / Image generation / Fast diffusion sampling
DreamCatalyst: Fast and High-Quality 3D Editing via Controlling Editability and Identity Preservation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DreamCatalyst, an SDS-based 3D editing method that reinterprets DDS as a single-step SDEdit reverse process, reweights identity preservation vs. editability over timesteps, and adds FreeU, yielding faster and higher-quality NeRF and 3DGS edits.
Problem
Existing SDS-based text-driven 3D editing methods are slow to train and produce low-quality results because their losses conflict with diffusion sampling dynamics and balance identity preservation against editability poorly.
Approach
Reinterprets the DDS objective as a single-step DDIM-based SDEdit denoising/renoising step, generalizes PDS with an explicit identity regularizer and timestep-dependent weights (identity weight increasing with t, editability weight rising with t), decreasing timestep sampling, and uses FreeU in the InstructPix2Pix backbone to boost editability without extra cost.
Key results
- DreamCatalyst fast mode is about 23x faster than PDS and high-quality mode about 8x faster, and high-quality mode is 1.85x faster than IN2N.
- On NeRF editing DreamCatalyst achieves the highest CLIP directional similarity, CLIP image similarity and aesthetic score (e.g., 0.171 / 0.744 / 5.564 vs PDS 0.161 / 0.687 / 5.437 in the FreeU ablation table).
- User study with 100 MTurk participants shows DreamCatalyst preferred by nearly three times as many selections as baselines across prompt alignment, quality and identity preservation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add FreeU (b=1.1) to DreamCatalyst (ablation) | DreamCatalyst (NeRF) · NeRF scenes (40 prompt pairs) | CLIP directional similarity: 0.171 -> 0.180 | Helped |
| add FreeU (b=1.1) to DreamCatalyst (ablation) | DreamCatalyst (NeRF) · NeRF scenes (40 prompt pairs) | Aesthetic score: 5.564 -> 5.688 | Helped |
| add FreeU (b=1.1) to PDS (ablation) | PDS (NeRF) · NeRF scenes (40 prompt pairs) | CLIP directional similarity: 0.161 -> 0.162 | Helped |
| add FreeU (b=1.1) to PDS (ablation) | PDS (NeRF) · NeRF scenes (40 prompt pairs) | CLIP image similarity: 0.687 -> 0.668 | Hurt |
| add FreeU (b=1.1) to PDS (ablation) | PDS (NeRF) · NeRF scenes (40 prompt pairs) | Aesthetic score: 5.437 -> 5.413 | Hurt |
| replace PDS loss with DreamCatalyst reweighted objective (no FreeU) and decreasing timestep sampling | DreamCatalyst vs PDS (NeRF) · NeRF scenes (40 prompt pairs) | CLIP image similarity: 0.687 -> 0.744 | Helped |
| use DreamCatalyst instead of PDS (fast mode, ~1000 iterations) | NeRF editing speed · NeRF scenes editing time | editing time speedup vs PDS: ~23x faster (fast mode); ~8x faster (high-quality mode) | Helped |
| use DreamCatalyst instead of IN2N (high-quality mode) | NeRF editing speed · NeRF scenes editing time | editing time speedup vs IN2N: 1.85x faster | Helped |
| apply FreeU with b=1.3 instead of b=1.1 in DreamCatalyst (ablation) | DreamCatalyst (NeRF) · NeRF scenes (40 prompt pairs) | CLIP directional similarity: increases with b, but other metrics decrease at b=1.3 | Mixed |
| apply DreamCatalyst specialized weighting Phi*/Psi* with decreasing timestep sampling in 3DGS (GaussianEditor combined) | GaussianEditor (3DGS) · 3DGS scenes | training speed: 1.25x training speedup | Helped |
Surprising findings
- Applying FreeU to PDS increased CLIP directional similarity only slightly (0.161 to 0.162) while lowering CLIP image similarity (0.687 to 0.668) and aesthetic score, indicating FreeU's benefit depends on the base method's identity preservation.
- Applying PDS with decreasing timestep scheduling degraded fine details and identity, so the diffusion-reverse-process schedule does not transfer directly to PDS.
Limitations
- Over-editing and over-smoothing occur when the FreeU parameter b is set too high (b=1.3).
- Relies on InstructPix2Pix and the quality of the source-scene initialization; 3DGS-specific methods use segmentation masks that can be inaccurate.
Experimental setup
Datasets: IN2N datasets PDS datasets
Benchmarks: IN2N scenes PDS scenes
Metrics: CLIP directional similarity CLIP image similarity aesthetic score editing time user study preference
Baselines: Instruct-NeRF2NeRF Posterior Distillation Sampling GaussianEditor DGE
Builds on: Posterior Distillation Sampling Delta Denoising Score SDEdit DreamSampler FreeU
Closest papers by meaning
- DaCapo: Score Distillation as Stacked Bridge for Fast and High-quality 3D Editing (CVPR 2025 · similarity 0.92)
- Rethinking Score Distilling Sampling for 3D Editing and Generation (ICML 2025 · similarity 0.91)
- DreamTime: An Improved Optimization Strategy for Diffusion-Guided 3D Generation (ICLR 2024 · similarity 0.89)
- ProEdit: Simple Progression is All You Need for High-Quality 3D Scene Editing (NeurIPS 2024 · similarity 0.89)
- Posterior Distillation Sampling (CVPR 2024 · similarity 0.89)
- ED-NeRF: Efficient Text-Guided Editing of 3D Scene With Latent Space NeRF (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.