Research map / Vision-language and multimodal / Multimodal LLMs
iFinder: Structured Zero-Shot Vision-Based LLM Grounding for Dash-Cam Video Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes iFinder, a training-free modular pipeline that extracts structured dash-cam cues (pose, lane, distance, orientation, trajectories) into a hierarchical JSON and prompts GPT-4o-mini with a peer V-VLM's answer for zero-shot driving video reasoning.
Problem
Post-hoc dash-cam driving video analysis with vision as the only modality is hard for end-to-end video VLMs, which struggle with spatial reasoning, causal inference, and explainability.
Approach
Decouple perception from reasoning: pretrained vision modules convert the video into a hierarchical, frame-indexed structured data format (ego state, lane assignments, metric distances, 3D yaw, attributes), which an LLM reasons over together with a peer V-VLM's answer, guided by a three-block prompt (key explanation, step instructions, peer instruction).
Key results
- On MM-AU multiple-choice VQA, iFinder reaches 63.39% accuracy versus 50.95% for VideoLLaMA2 and 24.22% for DriveMM.
- On LingoQA, iFinder achieves 44.20 Lingo-Judge accuracy, versus 41.20 for VideoChat2 and 13.40 for WiseAD.
- On Nexar accident prediction, iFinder reaches 62.0% accuracy versus 58.0% for VideoChat2 as peer model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add 3D object orientation estimation (yaw) (ablation) | iFinder · MM-AU | Accuracy (%): 63.39 vs 58.83 without (+4.56) | Helped |
| add scene understanding (ablation) | iFinder · MM-AU | Accuracy (%): +5.58 (57.81 without) | Helped |
| add distance estimation (ablation) | iFinder · MM-AU | Accuracy (%): +2.77 (60.62 without) | Helped |
| add lane detection (ablation) | iFinder · MM-AU | Accuracy (%): +1.59 (61.80 without) | Helped |
| add frame undistortion (ablation) | iFinder · MM-AU | Accuracy (%): +2.92 (60.47 without) | Helped |
| add ego-state estimation (ablation) | iFinder · MM-AU | Accuracy (%): +1.02 (62.37 without) | Helped |
| add object attributes (color) (ablation) | iFinder · MM-AU | Accuracy (%): +4.35 (59.04 without) | Helped |
| add key explanation prompt block (ablation) | iFinder · MM-AU | Accuracy (%): +4.66 (58.73 without) | Helped |
| add step instruction prompt block (ablation) | iFinder · MM-AU | Accuracy (%): +3.33 (60.06 without) | Helped |
| add peer instruction prompt block (ablation) | iFinder · MM-AU | Accuracy (%): +2.77 (60.62 without) | Helped |
| apply structured grounding pipeline vs end-to-end V-VLMs (no fine-tuning) | iFinder · MM-AU and SUTD | Accuracy (%): MM-AU 63.39 vs best generalist 52.89 (VideoLLaMA2 w/ pt); SUTD 50.93 vs 47.51 | Helped |
| use structured grounding pipeline vs WiseAD/driving-specialized models | iFinder · LingoQA | Lingo-Judge accuracy: 44.20 vs 13.40 (WiseAD) | Helped |
Surprising findings
- Removing object orientation estimation caused the largest ablation drop (63.39 to 58.83), larger than removing distance or lane location.
- Even when only 0.68% of objects are retained at a 0.8 confidence threshold, accuracy stays at 58.27%, above the best baseline.
Limitations
- Struggles with visually ambiguous collisions where subtle physical contact leaves no visual trace.
- Lacks mechanisms for intent, yielding, and social or normative aspects of driving scenes.
Experimental setup
Datasets: MM-AU SUTD-TrafficQA LingoQA Nexar
Benchmarks: MM-AU SUTD-TrafficQA LingoQA Nexar
Metrics: accuracy Lingo-Judge accuracy F1-score BLEU METEOR CIDEr
Baselines: VideoLLaMA2 VideoChat2 Video-LLaVA DriveMM WiseAD
Builds on: GeoCalib DROID-SLAM OWL-V2 Metric3D SAM
Closest papers by meaning
- Motion-Grounded Video Reasoning: Understanding and Perceiving Motion at Pixel Level (CVPR 2025 · similarity 0.87)
- Think Before You Drive: World Model-Inspired Multimodal Grounding (CVPR 2026 · similarity 0.86)
- TUMTraf VideoQA: Dataset and Benchmark for Unified Spatio-Temporal Video Understanding in Traffic Scenes (ICML 2025 · similarity 0.86)
- STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Driving (NeurIPS 2025 · similarity 0.86)
- ROVER: Recursive Reasoning Over Videos with Vision-Language Models for Embodied Tasks (NeurIPS 2025 · similarity 0.86)
- GroundVLP: Harnessing Zero-Shot Visual Grounding from Vision-Language Pre-training and Open-Vocabulary Object Detection (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.