Research map / Vision-language and multimodal / Human and embodied modeling
The Scene Language: Representing Scenes with Programs, Words, and Embeddings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the Scene Language, a representation of scenes as a program, words, and CLIP embeddings, inferred training-free from text or image inputs with an LLM and rendered into 3D/4D scenes.
Problem
Existing scene representations (scene graphs, program-based methods, diffusion latents) lack a joint treatment of hierarchical structure, semantic class, and instance-level visual identity, limiting fidelity and precise editing in 3D/4D scene generation.
Approach
A scene is represented as a triple of a Lisp-style entity-function program, natural-language words binding each function to a semantic class, and an ordered list of neural (CLIP text-space) embeddings giving each entity's identity; a pre-trained LM generates the program and words without training, and embeddings are obtained from text via CLIP or from images via textual inversion.
Key results
- Text-prompted 3D generation: Ours reaches 85.65 user-preference alignment on numeric prompts vs 10.79 for MVDream and 3.56 for GraphDreamer, with counting accuracy 1.0 vs 0.11.
- Image-prompted generation: LPIPS of 0.681 vs 0.811 for GraphDreamer (lower is better).
- Text-prompted 4D generation: 4D-fy achieves higher CLIP similarity (0.352 vs 0.341) while Ours has larger motion (5.9% vs 0.2% dynamic degrees).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the Scene Language program representation instead of direct scene generation (ablation) | MVDream · Numeric prompts | Count: 0.11 -> 1.0 | Helped |
| Use the Scene Language instead of MVDream | Text-prompted 3D generation · Numeric prompts | Alignment: 10.79±12.83 -> 85.65±13.71 | Helped |
| Use the Scene Language instead of MVDream | Text-prompted 3D generation · Numeric prompts | CLIP: 0.312±0.046 -> 0.351±0.051 | Helped |
| Use the Scene Language instead of GraphDreamer | Text-prompted 3D generation · Generic prompts | Alignment: 15.75±20.39 -> 52.13±26.83 | Helped |
| Use the Scene Language instead of GraphDreamer | Text-prompted 3D generation · Generic prompts | CLIP: 0.230±0.097 -> 0.328±0.030 | Mixed |
| Use the Scene Language instead of GraphDreamer | Image-prompted scene generation · Image-prompted scene generation | LPIPS: 0.811 -> 0.681 | Helped |
| Use the Scene Language instead of 4D-fy | Text-prompted 4D generation · Text-prompted 4D generation | CLIP: 0.352 -> 0.341 | Hurt |
| Use the Scene Language instead of 4D-fy | Text-prompted 4D generation · Text-prompted 4D generation | Dynamic: 0.2% -> 5.9% | Helped |
Surprising findings
- On generic prompts, MVDream has higher user-preference alignment than the Scene Language (35.13 vs 52.13 is in favor of ours; but CLIP scores are tied at 0.328), so the advantage is concentrated on numeric prompts.
- 4D-fy has higher per-frame CLIP alignment than the proposed method in 4D generation, though the proposed method produces much more motion.
Limitations
- 4D-fy achieves higher prompt alignment in 4D generation.
- Rendering with SDS-based Gaussian splatting depends on guidance from MVDream and ControlNet, and inference relies on a closed-source LM (Claude 3.5 Sonnet).
- Generic-prompt results show smaller gains over MVDream than numeric-prompt results.
Experimental setup
Metrics: user preference alignment CLIP similarity counting accuracy Dynamic degrees LPIPS
Baselines: GraphDreamer MVDream 4D-fy
Builds on: GraphDreamer MVDream Textual Inversion GALA3D 3D Gaussian Splatting
Closest papers by meaning
- Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understanding (NeurIPS 2024 · similarity 0.90)
- From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models (CVPR 2024 · similarity 0.89)
- OpenLex3D: A Tiered Benchmark for Open-Vocabulary 3D Scene Representations (NeurIPS 2025 · similarity 0.89)
- Scene Graph Generation with Role-Playing Large Language Models (NeurIPS 2024 · similarity 0.89)
- SceneScaffold: Active Scene-State Construction for Unified 3D Scene Understanding (NeurIPS 2026 · similarity 0.89)
- Universal Scene Graph Generation (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.