Research map / 3D vision / NeRF and radiance fields
LiveScene: Language Embedding Interactive Radiance Fields for Physical Scene Control and Rendering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LiveScene, a scene-level language-embedded interactive radiance field that factorizes multi-object interaction space into local 4D deformable feature planes with interaction-aware CLIP language embeddings, plus OmniSim and InterReal datasets.
Problem
Reconstructing and controlling multiple interactive articulated objects in complex scenes from monocular video, with natural-language grounding that remains consistent under topology changes caused by interactions; existing methods scale poorly in memory and lack language interfaces.
Approach
Decompose the (3+α)-dimensional interactive space into local deformable fields via a learned interaction probability decoder over multi-scale K-Planes feature planes, and store language embeddings indexed by maximum-probability interaction variables so embeddings adapt to object state changes.
Key results
- On CoNeRF Synthetic, LiveScene reaches 43.349 PSNR, outperforming the second-best method by 9.894 dB.
- On OmniSim, LiveScene averages 33.158 PSNR and 86.86 mIOU for language grounding, versus 21.74 mIOU for LERF.
- LiveScene keeps a constant ~39M parameters as interactive object count grows, while maintaining the best PSNR on the OmniSim #challenging subset (31.645).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LiveScene instead of the strongest listed baseline, CoGS | CoNeRF Synthetic novel-view synthesis · CoNeRF Synthetic | PSNR: +9.894 | Helped |
| Use LiveScene instead of CoGS | CoNeRF Controllable · CoNeRF Controllable | PSNR: 32.601 -> 32.782 | Helped |
| Use LiveScene instead of CoNeRF | novel-view synthesis · OmniSim #challenging | PSNR: +1.30 | Helped |
| Use LiveScene instead of CoNeRF | novel-view synthesis · InterReal #challenging | PSNR: +1.99 | Helped |
| Use LiveScene instead of LERF | language grounding · OmniSim | mIOU: +65.12 | Helped |
| Add multi-scale factorization (ablation) | LiveScene rendering · OmniSim ablation subset | PSNR: 31.740 -> 35.094 | Helped |
| Add multi-scale factorization (ablation) | LiveScene rendering · InterReal ablation subset | PSNR: 25.329 -> 28.289 | Helped |
| Add a learnable variable for each frame (ablation) | LiveScene · InterReal ablation subset | PSNR: 28.289 -> 29.577 | Helped |
| Add feature repulsion loss (ablation) | LiveScene · InterReal ablation subset | PSNR: 29.577 -> 29.959 | Helped |
| Add probability rejection (ablation) | LiveScene · InterReal ablation subset | PSNR: 29.577 -> 30.123 | Helped |
| Add maximum-probability embedding retrieval (ablation) | LiveScene language grounding · InterReal ablation subset | mIOU: 93.10 -> 30.40 | Hurt |
| Add interaction-aware language embedding (ablation) | LiveScene language grounding · OmniSim ablation subset | mIOU: 71.64 -> 78.52 | Helped |
Surprising findings
- On CoNeRF Controllable, LiveScene has lower SSIM (0.932) than several baselines despite the best PSNR (32.782).
- SAM and LERF are reported as having poor grounding under interaction-induced topology changes, with LERF at 21.74 average mIOU.
Limitations
- Control ability is limited by label density.
- Natural language control is restricted to a closed vocabulary tied to the underlying CLIP/OpenCLIP model.
- Per-object masks and state supervision are needed; ground-truth states only available on OmniSim.
Experimental setup
Datasets: OmniSim InterReal CoNeRF Synthetic CoNeRF Controllable
Benchmarks: CoNeRF Synthetic CoNeRF Controllable OmniSim InterReal
Metrics: PSNR SSIM LPIPS mIOU Depth L1
Baselines: NeRF InstantNGP 3DGS HyperNeRF K-Planes CoNeRF CoGS MK-Planes LERF SAM
Builds on: K-Planes CoNeRF CoGS LERF Nerfstudio
Closest papers by meaning
- The Scene Language: Representing Scenes with Programs, Words, and Embeddings (CVPR 2025 · similarity 0.88)
- Rethinking Open-Vocabulary Segmentation of Radiance Fields in 3D Space (AAAI 2025 · similarity 0.87)
- SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent (NeurIPS 2025 · similarity 0.87)
- IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Rendering (NeurIPS 2025 · similarity 0.86)
- SceneFun3D: Fine-Grained Functionality and Affordance Understanding in 3D Scenes (CVPR 2024 · similarity 0.86)
- InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.