Research map / 3D vision / 3D generation
Track, Inpaint, Resplat: Subject-driven 3D and 4D Generation with Progressive Texture Infilling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TIRE (Track, Inpaint, Resplat), which uses backward video tracking, progressive subject-driven LoRA inpainting of Stable Diffusion, and mask-aware multi-view refinement before Gaussian resplatting to make generated 3D/4D assets preserve a subject's identity.
Problem
Existing 3D and 4D generation methods fail to preserve the identity of a specific subject across unobserved viewpoints when given one or few reference images.
Approach
Treat subject-driven 3D/4D generation as 2D problems: use backward video tracking (CoTracker) to find occluded regions in rendered views, progressively inpaint them with a RealFill-style LoRA-personalized Stable Diffusion inpainting model starting from a 20-degree anchor and expanding outward, then resplat the infilled multi-view observations to 3D with LGM after mask-aware multi-view denoising.
Key results
- On DreamBooth-Dynamic DINO ViT-S/16 similarity, TIRE scores 0.5665 vs L4GM 0.5506, while Customize-It-3D ranks highest at 0.5773, which the authors attribute to DINO's inability to capture geometry.
- Under the averaged six-VLM subject-consistency score, TIRE achieves 1.854 vs Hunyuan3D-v2.5 1.703 and TRELLIS 1.476.
- On in-the-wild L4GM data, TIRE on L4GM improves the average VLM score from 2.346 to 2.371.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply TIRE instead of Hunyuan3D-v2.5 | Hunyuan3D-v2.5 · DreamBooth-Dynamic 3D | Average VLM score: 1.703 -> 1.854 | Helped |
| Apply TIRE instead of TRELLIS | TRELLIS · DreamBooth-Dynamic 3D | Average VLM score: 1.476 -> 1.854 | Helped |
| Apply TIRE instead of L4GM | L4GM · DreamBooth-Dynamic 4D | DINO (ViT-S/16): 0.5506 -> 0.5665 | Helped |
| Apply TIRE instead of STAG4D | STAG4D · DreamBooth-Dynamic 4D | DINO (ViT-S/16): 0.5287 -> 0.5665 | Helped |
| Apply TIRE instead of Customize-It-3D | Customize-It-3D · DreamBooth-Dynamic 4D | DINO (ViT-S/16): 0.5773 -> 0.5665 | Hurt |
| Apply TIRE instead of Hunyuan3D-v2.5 | Hunyuan3D-v2.5 · DreamBooth-Dynamic 3D | VLM score (Gemma 3 27B): 2.098 -> 2.103 | No clear effect |
| Add progressive inpainting (ablation) | TIRE · DreamBooth-Dynamic 3D | identity preservation | Helped |
| Use the 30% denoising schedule rather than 15% (ablation) | TIRE inpainting · DreamBooth-Dynamic 3D | inpainting quality | Helped |
| Use the 30% denoising schedule rather than 50% (ablation) | TIRE inpainting · DreamBooth-Dynamic 3D | inpainting quality | Helped |
| Use 10° instead of 20° progressiveness (ablation) | TIRE inpainting · DreamBooth-Dynamic 3D | inpainting quality | No clear effect |
| Use 30° instead of 20° progressiveness (ablation) | TIRE inpainting · DreamBooth-Dynamic 3D | inpainting quality | Hurt |
| Add the inpainting process (ablation) | TIRE · DreamBooth-Dynamic 3D | identity preservation | Helped |
Surprising findings
- DINO similarity ranks Customize-It-3D highest even though qualitative results show it performs worse, suggesting DINO is unsuitable for subject-driven 3D/4D evaluation.
- Modern native 3D generators such as TRELLIS and Hunyuan3D-v2.5 still show weak identity preservation, with VLM scores far from the maximum of 4.
Limitations
- Runtime of about 100 minutes per subject on a single A100 is slow, inherited from RealFill.
- DINO-based and VLM-based metrics are imperfect for judging geometry and identity in 3D/4D.
- 4D handling is essentially per-frame 3D generation with no temporal-specific subject-driven operations.
Experimental setup
Datasets: DreamBooth DreamBooth-Dynamic
Benchmarks: DreamBooth-Dynamic
Metrics: DINO ViT-S/16 similarity DINO ViT-B/16 similarity VLM-based subject consistency score (0-4) user study score (1-10)
Baselines: Customize-It-3D SV4D STAG4D L4GM TRELLIS Hunyuan3D-v2.5 Wonder3D SV3D MeshFormer
Builds on: RealFill LGM CoTracker DreamBooth
Closest papers by meaning
- NeRFiller: Completing Scenes via Generative 3D Inpainting (CVPR 2024 · similarity 0.86)
- 4Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Models (NeurIPS 2024 · similarity 0.86)
- A Unified Approach for Text- and Image-guided 4D Scene Generation (CVPR 2024 · similarity 0.86)
- ArtiScene: Language-Driven Artistic 3D Scene Generation Through Image Intermediary (CVPR 2025 · similarity 0.86)
- InstaInpaint: Instant 3D-Scene Inpainting with Masked Large Reconstruction Model (NeurIPS 2025 · similarity 0.86)
- Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpainting (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.