Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Feedback-Guided Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FeD, an LLaVA-based end-to-end sensorimotor driving agent trained by distillation from a BEV privileged agent and fine-tuned with auto-generated language feedback on prediction failures, with waypoints predicted in one pass from token embeddings.
Problem
Behavior-cloning driving policies are brittle and generalize poorly; existing LLM-based driving methods rely on closed-source GPT APIs, slow autoregressive generation, and cannot be trained end-to-end or evaluated closed-loop.
Approach
Train an open MLLM (LLaVA-7B with LoRA) to predict waypoints directly from waypoint-token hidden states, distill features from a privileged BEV-informed agent, and fine-tune with rule-generated language feedback describing collisions, traffic-light violations and route/expert deviations so the model learns to correct its own failures.
Key results
- FeD achieves 81.60 Driving Score on CARLA, 16.6% above the best LiDAR-based TransFuser++ (70.00) using only a front camera.
- On nuScenes, FeD reaches average L2 of 0.34 m and collision rate of 0.06% under ST-P3 metrics, improving L2 by 8.1% over VAD and reducing collisions versus GPT-Driver.
- Feedback fine-tuning improves Driving Score from 69.14 (w/o feedback, distillation) to 80.51 for the privileged-prompt ablation and boosts FeD DS by 12.6%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FeD instead of TransFuser++ | CARLA driving agent · CARLA LAV | Driving Score: 70.00 -> 81.60 | Helped |
| Use FeD instead of VAD | nuScenes driving agent · nuScenes val, ST-P3 metrics, average | L2 displacement error: 0.37 -> 0.34 | Helped |
| Use FeD instead of GPT-Driver | nuScenes driving agent · nuScenes val, ST-P3 metrics, average | Collision rate: 0.17 -> 0.06 | Helped |
| Use FeD instead of GPT-Driver | nuScenes driving agent · nuScenes val, UniAD metrics, average | L2 displacement error: 0.84 -> 0.58 | Helped |
| Use FeD instead of GPT-Driver | nuScenes driving agent · nuScenes val, UniAD metrics, average | Collision rate: 0.44 -> 0.19 | Helped |
| Add language-based feedback fine-tuning (ablation) | FeD sensorimotor agent · CARLA LAV | Driving Score: 72.45 -> 81.60 | Helped |
| Add privileged environmental information and feature distillation (ablation) | FeD sensorimotor agent · CARLA LAV | Driving Score: 69.14 -> 80.51 | Helped |
| Use a TinyLLaVA-based smaller model (ablation) | FeD · CARLA LAV | Driving Score: 81.60 -> 74.65 | Hurt |
Surprising findings
- The feedback-tuned sensorimotor agent achieves a higher Driving Score (81.60) than its privileged teacher (80.51).
Limitations
- Real-time deployment is still limited by LLM computational overhead.
- Fine-tuning with feedback requires hundreds of iterations and small batch size, so sample efficiency is low.
- Feedback taxonomy is rule-based and limited to five failure types; coarse high-level feedback is not explored.
Experimental setup
Datasets: CARLA (LAV benchmark training data) nuScenes
Benchmarks: CARLA LAV benchmark nuScenes validation
Metrics: Driving Score Route Completion Infraction Score L2 displacement error collision rate
Baselines: NEAT LAV TransFuser InterFuser TCP CaT TransFuser++ ST-P3 VAD GPT-Driver UniAD
Builds on: LLaVA LAV TransFuser GPT-Driver
Closest papers by meaning
- Prioritizing Perception-Guided Self-Supervision: A New Paradigm for Causal Modeling in End-to-End Autonomous Driving (NeurIPS 2025 · similarity 0.88)
- LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs (CVPR 2024 · similarity 0.87)
- DriveGPT4-V2: Harnessing Large Language Model Capabilities for Enhanced Closed-Loop Autonomous Driving (CVPR 2025 · similarity 0.87)
- LMDrive: Closed-Loop End-to-End Driving with Large Language Models (CVPR 2024 · similarity 0.87)
- DriveAgent-R1: Advancing VLM-based Autonomous Driving with Active Perception and Hybrid Thinking (ICLR 2026 · similarity 0.87)
- Less Language, More Latents: Annotation-Efficient VLAs for Driving (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.