Research map / 3D vision / 3D generation
ReplaceAnything3D: Text-Guided Object Replacement in 3D Scenes with Compositional Scene Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RAM3D, a text-guided 3D object replacement method that erases an object via LangSAM masks and 3D inpainting distilled with HiFA, then synthesizes a replacement object composited into a Bubble-NeRF scene.
Problem
Text-driven replacement or addition of objects in 3D scenes while keeping multi-view consistency, lighting and shadow integration, and seamless blending with the surrounding scene.
Approach
An Erase-and-Replace pipeline: first inpaint background behind the removed object with a NeRF distilled from a text-to-image inpainting LDM (HiFA loss, halo-region reconstruction supervision), then render a separate foreground NeRF for the new object composited over the background, using a Bubble-NeRF localized region.
Key results
- On GARDEN, FACE and FERN prompts, RAM3D achieves higher CLIP text-image direction similarity than GaussianEditor and BlendedNeRF (e.g., Pineapple 0.2246 vs -0.0631).
- RAM3D has the lowest warping error across all listed prompts and datasets (e.g., Pineapple 1.26 vs 1.46 ×10^-2 for GaussianEditor).
- In a 15-participant user study, RAM3D was preferred for prompt matching and visual quality in most prompts (e.g., Chess 1.000 vs 0.000 for prompt matching).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Halo-region supervision during the Erase stage (ablation) | RAM3D · STATUE erase task | CLIP Sim.: 0.020 -> 0.145 | Helped |
| add depth loss during the Erase stage (ablation) | RAM3D · STATUE erase task | CLIP Sim.: 0.087 -> 0.145 | Helped |
| use HiFA loss instead of simple SDS loss (ablation) | RAM3D · STATUE replace task (A corgi on a white plinth) | CLIP Sim.: 0.217 -> 0.232 | Helped |
| use separate Erase and Replace stages instead of one-stage training (ablation) | RAM3D · STATUE replace task (A corgi on a white plinth) | CLIP Sim.: 0.218 -> 0.232 | Helped |
| add random background augmentation during the Replace stage (ablation) | RAM3D · STATUE replace task (A corgi on a white plinth) | CLIP Sim.: 0.214 -> 0.232 | Helped |
| add random background augmentation during the Replace stage (ablation) | RAM3D · STATUE object replacement | foreground/background separation quality | Helped |
| use separate compositional foreground/background fields instead of a monolithic NeRF (ablation) | RAM3D · STATUE object replacement | background reconstruction quality | Helped |
Surprising findings
- Blended-NeRF, which directly optimizes CLIP similarity to the target prompt, still scores lower than RAM3D on CLIP direction similarity.
- Instruct-NeRF2NeRF sometimes scores higher on CLIP direction consistency than RAM3D, because failed edits leave a solid-colour background that boosts the temporal metric.
Limitations
- Multi-object prompts can yield implausible compositions, and the multi-face Janus problem persists from text-to-image distillation.
- Erase-and-Replace may remove structural information, so it is unsuitable for editing object properties such as appearance or geometry.
- Quantitative evaluation is limited, with mainly qualitative results and a small user study of 15 KAIST-affiliated participants.
Experimental setup
Datasets: SPIn-NeRF STATUE SPIn-NeRF RED-NET NeRF FERN Mip-NeRF 360 GARDEN Instruct-NeRF2NeRF FACE
Benchmarks: GARDEN FACE FERN
Metrics: CLIP Text-Image Direction Similarity warping error (optical flow) CLIP Direction Consistency user preference
Baselines: Gaussian Editor Blended-NeRF Reference-Guided Inpainting (Mirzaei et al.) Instruct-NeRF2NeRF RePaint-NeRF DreamEditor
Builds on: HiFA DreamFusion Instruct-NeRF2NeRF LangSAM Instant-NGP
Closest papers by meaning
- Towards Text-guided 3D Scene Composition (CVPR 2024 · similarity 0.87)
- Disentangled 3D Scene Generation with Layout Learning (ICML 2024 · similarity 0.87)
- Progressive3D: Progressively Local Editing for Text-to-3D Content Creation with Complex Semantic Prompts (ICLR 2024 · similarity 0.86)
- A3D: Does Diffusion Dream about 3D Alignment? (ICLR 2025 · similarity 0.86)
- PartGen: Part-level 3D Generation and Reconstruction with Multi-view Diffusion Models (CVPR 2025 · similarity 0.85)
- SceneFactor: Factored Latent 3D Diffusion for Controllable 3D Scene Generation (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.