Research map / Vision-language and multimodal / Human and embodied modeling
BEV-TSR: Text-Scene Retrieval in BEV Space for Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BEV-TSR, which retrieves driving scenes in BEV space from text queries using an LLM text encoder with knowledge-graph prompting, shared cross-modal embeddings, and caption generation, and introduces the nuScenes-Retrieval benchmark.
Problem
Text-scene retrieval for autonomous driving data: existing image-text retrieval methods lack global scene representation and handle complex, lengthy driving descriptions poorly, and no well-formed retrieval dataset exists.
Approach
Retrieve driving scenes in BEV space instead of image space, encode text with an LLM enriched by knowledge-graph embeddings, and align the two modalities via a set of shared learnable embeddings (Q-Former-inspired) plus an auxiliary caption generation loss.
Key results
- BEV-TSR achieves 85.78% text retrieval and 87.66% scene retrieval top-1 accuracy on nuScenes-Retrieval Easy, outperforming CLIP-ViT-Base, SigLIP-Base, and EVA02-Base by a large margin.
- On nuScenes-Retrieval Hard, BEV-TSR reaches 66.08% text-retrieval R@1 and 67.90% scene-retrieval R@1 versus about 29-32% for image-space baselines.
- Ablation: moving retrieval from image space to BEV space raises scene R@1 from 46.44% to 81.94%; LoRA fine-tuning of Llama2 adds about 11.6 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use BEV-space retrieval instead of image-space retrieval (ablation) | BEV-TSR · nuScenes-Retrieval Easy | Scene retrieval R@1: 0.4644 -> 0.8194 | Helped |
| add Shared Cross-modal Embedding (SCE) (ablation) | BEV-TSR with BEV · nuScenes-Retrieval Easy | Scene retrieval R@1: 0.8194 -> 0.8431 | Helped |
| add Knowledge Graph Prompting (KGP) (ablation) | BEV-TSR with BEV and SCE · nuScenes-Retrieval Easy | Scene retrieval R@1: 0.8431 -> 0.8557 | Helped |
| add Caption Generation (CG) (ablation) | BEV-TSR with BEV, SCE, and KGP · nuScenes-Retrieval Easy | Scene retrieval R@1: 0.8557 -> 0.8766 | Helped |
| use BEVFormer instead of BEVDet (ablation) | BEV-TSR · nuScenes-Retrieval Easy | Scene retrieval R@1: 0.8021 -> 0.8766 | Helped |
| use Llama2 instead of BERT (ablation) | BEV-TSR · nuScenes-Retrieval Easy | Scene retrieval R@1: 0.5594 -> 0.7030 | Helped |
| fine-tune Llama2 with LoRA (ablation) | BEV-TSR · nuScenes-Retrieval Easy | Scene retrieval R@1: 0.7030 -> 0.8194 | Helped |
| use DistMult rather than TransE or ConvE for KGE (ablation) | BEV-TSR · nuScenes-Retrieval Easy | Scene retrieval R@1: 0.8559; 0.8602 -> 0.8766 | Helped |
| vary SCE shared embedding count among k=1024, k=2048, and k=4096 (ablation) | BEV-TSR · nuScenes-Retrieval Easy | Text retrieval R@1: 0.8487; 0.8473; 0.8578 | Mixed |
| vary shared embedding dimension among d=512, d=1024, and d=2048 (ablation) | BEV-TSR · nuScenes-Retrieval Easy | Text retrieval R@1: 0.8386; 0.8578; 0.8530 | Mixed |
| use BEV-TSR rather than the strongest listed baseline | nuScenes-Retrieval Easy · nuScenes-Retrieval Easy | Text retrieval R@1: 0.4846 -> 0.8578 | Helped |
| use BEV-TSR rather than the strongest listed baseline | nuScenes-Retrieval Easy · nuScenes-Retrieval Easy | Scene retrieval R@1: 0.5585 -> 0.8766 | Helped |
Surprising findings
- Reported BEV-TSR results on the Easy split are only moderately comparable to baselines since baselines were extended to six surrounding views by the authors.
Limitations
- Focuses on camera image representations; LiDAR and fusion are not evaluated.
- Evaluated only on nuScenes-derived retrieval data built from templated captions and QA pairs.
- Knowledge graph embedding relies on manual synonym mapping of keywords.
Experimental setup
Datasets: nuScenes NuScenes-QA ADKG (autonomous driving knowledge graph)
Benchmarks: nuScenes-Retrieval Easy nuScenes-Retrieval Hard
Metrics: R@1 R@5 R@10
Baselines: CLIP-ViT-Base SigLIP-Base EVA02-Base BEVDet BERT
Builds on: BEVFormer BLIP-2 Q-Former TransE DistMult NuScenes-QA
Closest papers by meaning
- Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Models (CVPR 2024 · similarity 0.86)
- VLM2Scene: Self-Supervised Image-Text-LiDAR Learning with Foundation Models for Autonomous Driving Scene Understanding (AAAI 2024 · similarity 0.86)
- Improving Bird's Eye View Semantic Segmentation by Task Decomposition (CVPR 2024 · similarity 0.85)
- M-BEV: Masked BEV Perception for Robust Autonomous Driving (AAAI 2024 · similarity 0.85)
- Driving Video Retrieval for Complex Queries with Structured Grounding (NeurIPS 2026 · similarity 0.85)
- NuScenes-QA: A Multi-Modal Visual Question Answering Benchmark for Autonomous Driving Scenario (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.