Research map / Video / Video-language understanding

Driving Video Retrieval for Complex Queries with Structured Grounding

NeurIPS 2026 · Video › Video-language understanding · approach: Text-video retrieval · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a data-calibrated, structured-grounding retrieval framework for driving videos that fits LLM-proposed symbolic rule thresholds via learning-to-rank under caption-based distant supervision, plus a confidence-gated adaptation policy and rank fusion with dense and BM25 retrieval.

Problem

Natural-language retrieval of dynamic driving events (e.g., cut-ins, hard braking) is missed by dense vision-language embeddings and keyword retrieval, while LLM-written symbolic rules have numerically miscalibrated thresholds that do not match the real perception-output distribution.

Approach

Keep the LLM responsible for rule structure but calibrate numerical constraints (thresholds, magnitudes, windows, weights) by maximizing average precision on auxiliary in-domain videos labeled weakly from auto-generated captions, using an LLM-as-optimizer loop; reuse calibrated rules for matched queries and synthesize exemplar-conditioned rules otherwise, then fuse with dense and BM25 via query-co

Key results

• On DrivingDojo, the method reaches 26.8% Acc@1 vs 14.6% for the strongest dense baseline (83.6% relative gain) and 8.6% mAP vs 4.7%.
• On MM-AU, Acc@1 improves from 20.4% to 26.5% over the best baseline (Qwen3-VL-8B).
• Average per-query latency is 3.10 s versus 4,868.05 s for NSVS-TL, about 1,500x faster.

Surprising findings

• Plain VLM binary classification per video performs poorly (Acc@1 2.4%, 7.3% with reranker), below embedding baselines.
• Substituting NSVS-TL into the same fusion pipeline lowers Acc@1 to 12.2%, below its standalone 22.0%.

Limitations

• Two LLM stages (rule synthesis and fusion) dominate latency and depend on GPT-4.1 API calls.
• Calibration relies on captions as noisy weak labels, and the approach depends on a perception pipeline with many pretrained modules.
• Evaluation uses small query sets (41 DrivingDojo queries) and the top-1 gains are modest in absolute terms.

Experimental setup

Datasets: Nexar DrivingDojo CarCrashDataset MM-AU

Benchmarks: DrivingDojo MM-AU CarCrashDataset

Metrics: MRR mAP Acc@1 Acc@3 Acc@5 Acc@10 latency

Baselines: Qwen3-VL-Embedding-2B Qwen3-VL-Embedding-8B SigLIP SigLIP2 VideoRAG NSVS-TL VLM binary classification

Builds on: NSVS-TL VideoRAG reciprocal rank fusion iFinder

Closest papers by meaning

• BEV-TSR: Text-Scene Retrieval in BEV Space for Autonomous Driving (AAAI 2025 · similarity 0.85)
• Less Language, More Latents: Annotation-Efficient VLAs for Driving (NeurIPS 2026 · similarity 0.85)
• SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Models (ICML 2025 · similarity 0.85)
• From Play to Replay: Composed Video Retrieval for Temporally Fine-Grained Videos (NeurIPS 2025 · similarity 0.84)
• DriveCombo: Benchmarking Compositional Traffic Rule Reasoning in Autonomous Driving (CVPR 2026 · similarity 0.84)
• iFinder: Structured Zero-Shot Vision-Based LLM Grounding for Dash-Cam Video Reasoning (NeurIPS 2025 · similarity 0.84)