Research map / 3D vision / 3D generation
Carve3D: Improving Multi-view Reconstruction Consistency for Diffusion Models with RL Finetuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Carve3D, an RL finetuning method (pure on-policy REINFORCE with KL regularization and scaling-law-guided batch size) that uses a NeRF-reconstruction-based multi-view consistency (MRC) reward to improve Instant3D's multi-view consistency.
Problem
Multi-view diffusion models finetuned with SFT for text-to-3D remain multi-view inconsistent, and longer SFT causes distribution shift that reduces diversity, texture detail and realism; there is no established quantitative multi-view consistency metric.
Approach
Define a Multi-view Reconstruction Consistency metric (LPIPS between generated views and sparse-view LRM NeRF renderings at the same cameras, with bounding-box normalization) and use its negative as the reward for on-policy policy-gradient RLFT of a multi-view diffusion model, with KL-advantage regularization to limit distribution shift.
Key results
- Carve3DM achieves avg MRC 0.0606 on the testing set vs 0.0892 for Instant3D-10K base and 0.0685 for Instant3D-100K.
- User study: 68.33% of 15 participants judged Carve3DM more 3D consistent than the base model.
- Scaling experiments show reward improves smoothly with compute (batch size) at optimal data size; RLFT trained on only 30 prompts generalizes to 415 test prompts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Carve3D RLFT to Instant3D-10K | Instant3D-10K · DreamFusion testing set | Avg MRC: 0.0892 -> 0.0606 | Helped |
| Increase SFT from 10K to 20K steps (ablation) | Instant3D · DreamFusion testing set | Avg MRC: 0.0892 -> 0.0795 | Helped |
| Increase SFT from 20K to 100K steps (ablation) | Instant3D · DreamFusion testing set | Avg MRC: 0.0795 -> 0.0685 | Helped |
| Replace partially on-policy importance-sampling training with pure on-policy policy gradient (ablation) | Carve3D RLFT · Testing set | training stability | Helped |
| Add normalized KL-divergence regularization (ablation) | Carve3D RLFT · Text-prompt generation | prompt alignment | Helped |
| Increase RLFT compute by increasing batch size under the optimal data size (ablation) | Carve3D RLFT · Multi-view consistency task | reward | Helped |
| Scale up prompt data beyond the optimal data size for a batch size (ablation) | Carve3D RLFT · Multi-view consistency task | reward | Hurt |
| Increase RLFT training epochs (ablation) | Carve3D RLFT · Multi-view consistency task | reward | Helped |
| Increase RLFT training epochs without stopping for KL divergence (ablation) | Carve3D RLFT · Text-prompt generation | prompt alignment | Hurt |
| Use Carve3D RLFT instead of the Instant3D base model | Instant3D-10K · User study | 3D consistency preference: 68.33% | Helped |
Surprising findings
- Pure on-policy REINFORCE was more stable than PPO-style DDPO_IS, contrary to common LLM RLHF practice.
- Even with a training prompt set of size 10, the model still generalized to test prompts.
Limitations
- Prompt alignment unchanged only with early stopping by KL threshold chosen empirically
- Improvement over Instant3D-100K is less obvious than over the 10K base
- Hypotheses for PPO instability are untested
Experimental setup
Datasets: Instant3D-10K Objaverse DreamFusion testing prompts
Benchmarks: DreamFusion prompt test set (415 prompts)
Metrics: MRC (LPIPS-based consistency, lower is better) user preference
Baselines: Instant3D-10K Instant3D-20K Instant3D-100K MVDream Zero123++ SyncDreamer
Builds on: DDPO REINFORCE Instant3D LRM
Closest papers by meaning
- Sparse3D: Distilling Multiview-Consistent Diffusion for Object Reconstruction from Sparse Views (AAAI 2024 · similarity 0.89)
- Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation (ICLR 2024 · similarity 0.88)
- Enhancing 3D Fidelity of Text-to-3D using Cross-View Correspondences (CVPR 2024 · similarity 0.88)
- SweetDreamer: Aligning Geometric Priors in 2D diffusion for Consistent Text-to-3D (ICLR 2024 · similarity 0.88)
- Efficient-3Dim: Learning a Generalizable Single-image Novel-view Synthesizer in One Day (ICLR 2024 · similarity 0.88)
- MVDream: Multi-view Diffusion for 3D Generation (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.