Research map / 3D vision / 3D generation
Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ChatSim, a system where collaborative GPT-4 LLM agents turn natural-language commands into editable, photo-realistic multi-camera 3D driving scene videos, using a new multi-camera NeRF (McNeRF) and lighting estimator (McLight) for rendering external 3D assets.
Problem
Editable scene simulation for autonomous driving is limited by inefficient user interaction (manual code-based editing), weak multi-camera photo-realistic rendering, and lack of support for external digital assets.
Approach
Decompose natural-language editing commands across specialized LLM agents (project manager, view adjustment, background rendering, vehicle deletion, asset management, vehicle motion, foreground rendering), paired with a multi-camera HDR NeRF (exposure-aware, multi-camera aligned) and a hybrid skydome plus NeRF-based surrounding lighting estimator for inserting 3D assets.
Key results
- McNeRF achieves PSNR 25.82 and LPIPS 0.378 on background novel view synthesis, beating F2NeRF (23.26 PSNR) and S-NeRF (24.71 PSNR) while inferring in 2.5 s.
- Multi-agent collaboration raises task completion accuracy on deletion from 0.617 to 0.983 and on abstract commands from 0.216 to 0.883.
- McLight reduces mean peak intensity log10 error to 0.449 from 0.590 for the best prior method (Wang et al.), and mean peak angular error to 32.3 degrees.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add multi-camera alignment to McNeRF (ablation) | McNeRF · Waymo Open Dataset novel view synthesis | PSNR: 23.32 -> 25.82 | Helped |
| Add multi-camera alignment to McNeRF (ablation) | McNeRF · Waymo Open Dataset novel view synthesis | SSIM: 0.776 -> 0.822 | Helped |
| Add multi-camera alignment to McNeRF (ablation) | McNeRF · Waymo Open Dataset novel view synthesis | LPIPS: 0.437 -> 0.378 | Helped |
| Add exposure-time modeling to McNeRF (ablation) | McNeRF · Waymo Open Dataset novel view synthesis | PSNR: 25.18 -> 25.82 | Helped |
| Add exposure-time modeling to McNeRF (ablation) | McNeRF · Waymo Open Dataset novel view synthesis | SSIM: 0.819 -> 0.822 | Helped |
| Add exposure-time modeling to McNeRF (ablation) | McNeRF · Waymo Open Dataset novel view synthesis | LPIPS: 0.381 -> 0.378 | Helped |
| Use collaborative multi-agent execution instead of a single LLM agent (ablation) | ChatSim · Language command task completion | Deletion accuracy: 0.617 -> 0.983 | Helped |
| Use collaborative multi-agent execution instead of a single LLM agent (ablation) | ChatSim · Language command task completion | Abstract command accuracy: 0.216 -> 0.883 | Helped |
| Use McLight instead of Wang et al. lighting estimation | Foreground rendering · HDRI dataset | Peak intensity (log10) error: 0.590 -> 0.449 | Helped |
| Use McLight instead of Wang et al. lighting estimation | Foreground rendering · HoliCity | Peak angular error (deg): 33.5 -> 32.3 | Helped |
| Use McLight instead of Wang et al. lighting estimation | Foreground rendering · Lighting estimation user study | User study preference (%): 37.3 -> 43.1 | Helped |
| Use the ChatSim text-to-motion method instead of GPT2Code | Vehicle motion generation · Text-to-motion evaluation | Within-road: 0.214 -> 1.000 | Helped |
Limitations
- Background editing is limited; weather changes and other background edits are not yet supported.
- Evaluation is on a small set of scenes (32 scenes for background rendering) and few commands.
- Relies on GPT-4 API for all agent LLM calls.
Experimental setup
Datasets: Waymo Open Dataset HoliCity 449 HDRIs collected from online HDRI databases
Benchmarks: Waymo Open Dataset
Metrics: PSNR SSIM LPIPS inference time peak intensity log10 error peak angular error user study task completion accuracy AP30/AP70 detection
Baselines: DVGO Mip-NeRF360 S-NeRF F2NeRF Hold-Geoffroy et al. lighting estimation Wang et al. lighting estimation GPT2Code GPT2Motion single LLM agent
Builds on: F2-NeRF Blender Lift-Splat Agisoft Metashape
Closest papers by meaning
- SceneCrafter: Controllable Multi-View Driving Scene Editing (CVPR 2025 · similarity 0.88)
- SceneDiffuser++: City-Scale Traffic Simulation via a Generative World Model (CVPR 2025 · similarity 0.85)
- Synthesizing Photorealistic and Dynamic Urban Environments for Multimodal Robot Navigation and Collaboration (NeurIPS 2025 · similarity 0.84)
- X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllability (NeurIPS 2025 · similarity 0.84)
- Generating Multimodal Driving Scenes via Next-Scene Prediction (CVPR 2025 · similarity 0.84)
- SceneCraft: An LLM Agent for Synthesizing 3D Scenes as Blender Code (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.