Research map / Large language models / Agents, code and math
DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiLu, a knowledge-driven autonomous driving framework where GPT-3.5 reasons over memory-recalled experiences and GPT-4 reflects on and corrects unsafe decisions, evaluated in Highway-env against an RL baseline.
Problem
Data-driven autonomous driving methods suffer from dataset bias, overfitting, poor generalization and limited interpretability; the paper asks how to instill human-like knowledge-driven decision-making into driving systems.
Approach
A driver agent built on LLMs with a Reasoning module (scenario description, few-shot retrieval from a vector memory, chain-of-thought decision), a Reflection module that summarizes safe/unsafe decisions and corrects mistakes with an LLM, and a persistent Memory module that accumulates experiences, so the system improves without weight updates.
Key results
- With 40 memory items, DiLu's 5-shot setting passes almost all tests, with median success steps over 25, while 0-shot median is below 5.
- In lane-4-density-2, DiLu reaches 70% success rate using 40 experiences versus 69% for GRAD after 600,000 training episodes; in lane-5-density-3 DiLu keeps 35% SR versus a 10% for GRAD.
- Memory built from CitySim real-world trajectories improves DiLu performance in Highway-env, and the reflection memory ablation raises median success steps from 10 to 24.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DiLu instead of GRAD | autonomous driving · lane-4-density-2 | success rate (SR): 69% -> 70% | Helped |
| use DiLu instead of GRAD | autonomous driving · lane-5-density-2.5 | success rate (SR): 47% -> 65% | Helped |
| use DiLu instead of GRAD | autonomous driving · lane-5-density-3 | success rate (SR): 10% -> 35% | Helped |
| increase few-shot experiences from 3-shot to 5-shot (ablation) | DiLu with 20 experiences · lane-4-density-2 | median success steps (SS): 13 -> 30 | Helped |
| increase few-shot experiences from 3-shot to 5-shot (ablation) | DiLu with 20 experiences · lane-5-density-3 | median success steps (SS): 5 -> 23 | Helped |
| add 12 success memories (ablation) | DiLu with 20 initial experiences · Highway-env | median success steps (SS): 10.0 -> 24.5 | Helped |
| add 6 correction memories (ablation) | DiLu with 20 initial experiences · Highway-env | median success steps (SS): 10.0 -> 21.5 | Helped |
| add both success and correction memories (ablation) | DiLu with 20 initial experiences · Highway-env | median success steps (SS): 10.0 -> 24.5 | Helped |
| use experiences extracted from CitySim instead of no prior experiences | DiLu · Highway-env | success steps (SS) | Helped |
Surprising findings
- Few-shot experiences need not be consistent in decisions; the authors note consistency is not required within the few-shot set.
Limitations
- Decision-making latency of 5-10 seconds from LLM inference and API response.
- Does not completely eliminate LLM hallucinations.
Experimental setup
Datasets: Highway-env simulator experiences CitySim
Benchmarks: Highway-env
Metrics: success steps success rate
Baselines: GRAD 0-shot DiLu
Builds on: Chain-of-thought prompting Voyager GRAD
Closest papers by meaning
- Hybrid-Driving: An Autonomous Driving Decision Framework Integrating Large Language Models, Knowledge Graphs and Driving Rules (AAAI 2025 · similarity 0.89)
- AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning (NeurIPS 2025 · similarity 0.88)
- LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs (CVPR 2024 · similarity 0.88)
- SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Models (ICML 2025 · similarity 0.87)
- Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learning (AAAI 2026 · similarity 0.87)
- World Knowledge-Enhanced Reasoning Using Instruction-Guided Interactor in Autonomous Driving (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.