Research map / Detection and segmentation / 3D detection and driving
Language Prompt for Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces NuPrompt, a language-prompt dataset expanding nuScenes with 40,147 object-referring descriptions over 3D multi-view multi-frame driving scenes, and PromptTrack, an end-to-end PF-Track-based tracker that grounds prompt-referred objects and predicts their trajectories.
Problem
Language-prompt-based object grounding in driving scenes is limited by the scarcity of paired prompt-instance data, and prior driving referring datasets are 2D, single-view, or scene-level QA rather than instance-level 3D tracking with language.
Approach
Builds an object-centric, instance-level language prompt dataset on nuScenes (language elements tagged to boxes, combined with AND, then rewritten by GPT-3.5 and filtered by annotators), and adds a prompt fusion module plus a prompt reasoning branch to PF-Track so that a single end-to-end tracker outputs only the prompt-referred tracks and their motion forecasts.
Key results
- PromptTrack with a PETR decoder reaches AMOTA 0.259, AMOTP 1.513, recall 36.6%, and IDS 26 on NuPrompt, the best among compared methods (ADA-Track 0.249 AMOTA).
- PromptTrack (PETR) achieves ADE 2.17 and FDE 2.21 over an 8-frame motion forecast horizon.
- Removing prompt reasoning (w/o Prompt Fusion) drops AMOTA to 0.073, and reducing prompts to 20% of training drops AMOTA to 0.210 from 0.259.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PromptTrack with PETR instead of ADA-Track with PETR | PromptTrack · NuPrompt validation | AMOTA: 0.249 -> 0.259 | Helped |
| Remove prompt fusion and use all tracking instances as prompt predictions (ablation) | PromptTrack · NuPrompt validation | AMOTA: 0.259 -> 0.073 | Hurt |
| Use two-way prompt-visual fusion (ablation) | PromptTrack · NuPrompt validation | AMOTA: 0.259 -> 0.249 | Hurt |
| Add prompt embedding to decoder queries (ablation) | PromptTrack · NuPrompt validation | AMOTA: 0.259 -> 0.054 | Hurt |
| Add past reasoning (ablation) | PromptTrack · NuPrompt validation | AMOTA: 0.239 -> 0.259 | Helped |
| Add future reasoning (ablation) | PromptTrack · NuPrompt validation | AMOTA: 0.247 -> 0.259 | Helped |
| Use unified descriptions instead of language elements as prompts (ablation) | PromptTrack · NuPrompt validation | AMOTA: 0.271 -> 0.242 | Hurt |
| Use total prompts instead of language elements as prompts (ablation) | PromptTrack · NuPrompt validation | AMOTA: 0.271 -> 0.259 | Hurt |
| Increase training prompt fraction from 20% to 50% (ablation) | PromptTrack · NuPrompt validation | AMOTA: 0.210 -> 0.247 | Helped |
| Increase training prompt fraction from 50% to 100% (ablation) | PromptTrack · NuPrompt validation | AMOTA: 0.247 -> 0.259 | Helped |
Surprising findings
- Training on unified (LLM-combined) descriptions yields lower AMOTA (0.242) than on individual language elements (0.271).
- The w/o Prompt Fusion variant achieves the highest AMOTP and recall (43.5%) despite very low AMOTA (0.073).
Limitations
- Unified LLM-generated descriptions are harder for the model than individual language elements.
- Relies on manual annotation and annotator filtering of LLM outputs, which limits scale and consistency.
Experimental setup
Datasets: nuScenes NuPrompt
Benchmarks: NuPrompt
Metrics: AMOTA AMOTP RECALL MOTA IDS ADE FDE
Baselines: CenterPoint tracker DQTrack ADA-Track
Builds on: PF-Track MOTR PETR DETR3D nuScenes
Closest papers by meaning
- Less Language, More Latents: Annotation-Efficient VLAs for Driving (NeurIPS 2026 · similarity 0.89)
- VLP: Vision Language Planning for Autonomous Driving (CVPR 2024 · similarity 0.88)
- Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models (NeurIPS 2025 · similarity 0.87)
- DriveSpatial: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving (NeurIPS 2026 · similarity 0.87)
- MAPLM: A Real-World Large-Scale Vision-Language Benchmark for Map and Traffic Scene Understanding (CVPR 2024 · similarity 0.87)
- Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Models (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.