Research map / Large language models / Agents, code and math
Continuously Learning, Adapting, and Improving: A Dual-Process Approach to Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LeapAD, a closed-loop CARLA driving system with a VLM for critical-object scene understanding and a dual-process (GPT-4 analytic / Qwen1.5-1.8B heuristic) decision module that improves via a reflection-driven memory bank.
Problem
Camera-only autonomous driving methods rely on large annotated datasets, generalize poorly to complex scenarios and causal reasoning, and LLM-based driving agents are mostly evaluated open-loop, so their closed-loop adaptability is unclear.
Approach
Combine a critical-object-focused VLM with a dual-process decision module in which a GPT-4 "analytic" LLM accumulates linguistic driving experience and reflects on accidents; the experience is distilled by SFT into a small "heuristic" model and used via retrieval-based few-shot prompting, enabling continual improvement without human labels.
Key results
- On CARLA Town05 Short, LeapAD reaches DS 83.11 with 11K annotations, above camera-only baselines such as Roach (65.26) and VAD (64.30) and above VLM + GPT-4 (81.31).
- The 1.8B heuristic model with an enlarged memory bank surpasses the GPT-4-powered analytic process, and performance rises with memory size (DS 51.77 at 0 samples to 72.33 at 9K in the Town05 ablation).
- Reported annotation count is 1-2 orders of magnitude smaller than camera/LiDAR baselines (e.g., InterFuser uses 3M annotations versus LeapAD's 11K).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Town05 experience to the memory bank and use reflection data (ablation) | LeapAD · Town05 Short | DS: 75.73±1.36 -> 83.11±0.28 | Helped |
| add few-shot prompting, moving from 0-shot to 1-shot (ablation) | LeapAD · Town05 Short | DS: 51.77 -> 64.95 | Helped |
| increase few-shot prompting from 1-shot to 2-shot (ablation) | LeapAD · Town05 Short | DS: 64.95 -> 71.20 | Helped |
| increase few-shot prompting from 2-shot to 3-shot (ablation) | LeapAD · Town05 Short | DS: 71.20 -> 72.33 | Helped |
| add a memory bank of 90 samples instead of no memory (ablation) | LeapAD few-shot system · Town05 Short | DS: 51.77 -> 60.72 | Helped |
| increase memory size from 90 to 900 samples (ablation) | LeapAD few-shot system · Town05 Short | DS: 60.72 -> 66.68 | Helped |
| increase memory size from 900 to 9000 samples (ablation) | LeapAD few-shot system · Town05 Short | DS: 66.68 -> 72.33 | Helped |
| add few-shot prompting to SFT using memory from Town01-04 and 06 (ablation) | LeapAD · Town05 | DS: 66.40 -> 75.73 | Helped |
| add few-shot prompting to SFT using Town05 memory (ablation) | LeapAD · Town05 | DS: 69.90 -> 78.07 | Helped |
| add reflection and incorporate corrected experiences into the memory bank (ablation) | LeapAD · Town05 | DS | Helped |
Surprising findings
- Zero-shot LeapAD without Town05 experience already beats several camera-only baselines, and few-shot retrieval adds further gains.
- On Town05 Long, LeapAD's IS is 51.7 with RC 100, so its driving score is far below the best LiDAR+camera methods.
Limitations
- Relies only on single-frame camera input with no temporal information.
- The VLM cannot take part in the reflection mechanism, which limits system-level improvement.
- Evaluation is limited to CARLA, with a gap to real-world agent behavior.
Experimental setup
Datasets: Rank2Tell DriveLM CARLA closed-loop collected data
Benchmarks: CARLA Town05 Short CARLA Town05 Long CARLA Town01 CARLA Town04
Metrics: Driving Score Route Completion Infraction Score
Baselines: InterFuser TransFuser VAD NEAT Roach WOR LBC CILRS
Builds on: dual-process theory Pure Pursuit Rank2Tell DriveLM
Closest papers by meaning
- Prioritizing Perception-Guided Self-Supervision: A New Paradigm for Causal Modeling in End-to-End Autonomous Driving (NeurIPS 2025 · similarity 0.89)
- DualAD: Disentangling the Dynamic and Static World for End-to-End Driving (CVPR 2024 · similarity 0.88)
- Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learning (AAAI 2026 · similarity 0.88)
- DriveAgent-R1: Advancing VLM-based Autonomous Driving with Active Perception and Hybrid Thinking (ICLR 2026 · similarity 0.88)
- nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Driving (NeurIPS 2026 · similarity 0.88)
- Hybrid-Driving: An Autonomous Driving Decision Framework Integrating Large Language Models, Knowledge Graphs and Driving Rules (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.