Research map / Large language models / Agents, code and math
ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehicles
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ChatScene, an LLM-based agent that turns text descriptions of safety-critical driving scenarios into executable Scenic/CARLA simulations via a retrieval database of Scenic code snippets, and uses the generated scenes to adversarially fine-tune RL ego vehicles.
Problem
Generating diverse, safety-critical driving scenarios for autonomous vehicle testing is limited by narrow hand-crafted or adversarial methods, and direct LLM code generation for Scenic often yields non-executable code or hallucinated APIs due to scarce training examples.
Approach
Decompose Scenic scenarios into behavior, geometry, and spawn-position components; build a retrieval database of description-to-code snippets (curated with GPT-4 and manual checks); have an LLM generate and decompose scenario descriptions, retrieve snippets via Sentence-T5 embeddings with faiss, and assemble executable Scenic scripts that are iteratively refined toward collisions.
Key results
- ChatScene achieves average collision rate 0.831 vs 0.676 for best baseline (CS) across eight base scenarios against three RL ego vehicles.
- Adversarial fine-tuning with ChatScene scenes reduces average collision rate to 0.043, versus 0.135 for the best baseline (AT) and 0.559 before fine-tuning.
- ChatScene attains the highest ADE (4.567) indicating greater scenario diversity than baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Generate adversarial scenarios with ChatScene rather than the strongest baseline | Scenario generation evaluated against RL-trained ego vehicles · 8 base scenarios (average) | collision rate: 0.676 -> 0.831; 15% increase | Helped |
| Fine-tune the ego vehicle using ChatScene-generated adversarial scenarios rather than the strongest baseline scenarios | SAC-trained ego vehicle · 8 base scenarios (average), post-finetuning test | collision rate: 0.135 -> 0.043; an additional 9% | Helped |
Surprising findings
- Directly prompting LLMs to write Scenic code often produces non-compilable code or nonexistent APIs, motivating the retrieval-based approach.
Limitations
- Evaluation restricted to SafeBench/CARLA with three RL ego vehicles and eight base scenarios.
- Snippet database requires manual evaluation and correction of GPT-4-generated code.
- Only the last three scenario components are retrieved; ego behavior is not defined.
Experimental setup
Datasets: Scenic CARLA examples CARLA Challenge base scenarios (NHTSA-derived)
Benchmarks: SafeBench CARLA Challenge eight base traffic scenarios
Metrics: collision rate overall score average displacement error route completion time spent acceleration yaw velocity lane invasion
Baselines: Learning-to-collide AdvSim Carla Scenario Generator Adversarial Trajectory Optimization
Builds on: Scenic SafeBench Sentence-T5 faiss
Closest papers by meaning
- Hybrid-Driving: An Autonomous Driving Decision Framework Integrating Large Language Models, Knowledge Graphs and Driving Rules (AAAI 2025 · similarity 0.89)
- SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Models (ICML 2025 · similarity 0.87)
- Generating Traffic Scenarios via In-Context Learning to Learn Better Motion Planner (AAAI 2025 · similarity 0.86)
- LLM-Powered Synthetic Environments for Self-Driving Scenarios (AAAI 2024 · similarity 0.86)
- Generative Scenario Rollouts for End-to-End Autonomous Driving (NeurIPS 2026 · similarity 0.85)
- HCRMP: An LLM-Hinted Contextual Reinforcement Learning Framework for Autonomous Driving (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.