Research map / Large language models / Agents, code and math
PSI: A Benchmark for Human Interpretation and Response in Traffic Interactions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PSI, a benchmark of 196 driver-view pedestrian encounter videos with continuous, multi-annotator crossing-intent labels, driver decisions, and textual reasoning, and evaluates VLMs and an explainable trajectory model (eP2P) on it.
Problem
Existing pedestrian crossing-intent datasets label intent at isolated critical frames, conflate intention types, ignore annotator subjectivity, and lack verbal reasoning, limiting models that predict and explain pedestrian and driver behavior for autonomous vehicles.
Approach
Continuous frame-by-frame pedestrian situated intent and driver decision labels from 9-24 diverse annotators per video, with soft labels (disagreement) and five-aspect textual reasoning attached to each segment change, collected via a Point-and-Explain annotation UI on naturalistic driving video.
Key results
- eP2P with disagreement and reasoning reaches 63.09% accuracy and 58.55 F1 on PSI intent prediction, versus 59.04% accuracy / 56.13 F1 for PIE.
- On trajectory prediction at 1.5s, eP2P achieves ADE 45.55 / FDE 97.00 pixels at 1.0s-horizon range, improving over BiTraP-NP (48.37 ADE at 1.5s)
- Off-the-shelf VLMs (Qwen-VL, InternVL, SmolVLM, LLaMA 3.2 Vision) achieve low CIDEr (~0.02) on PSI reasoning generation, and models often fail to detect pedestrians or hallucinate.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fuse global and local contextual knowledge | eP2P versus PIE · PSI intent prediction | F1: 56.13% to 57.01% | Helped |
| Add disagreement-based objective reweighting (ablation) | eP2P · PSI intent prediction | Acc: 60.30 -> 61.67 | Helped |
| Add reasoning generator and train with textual annotations (ablation) | eP2P+disagr. · PSI intent prediction | Acc: 61.67 -> 63.09 | Helped |
| Add evidential uncertainty prediction layer and corresponding training objectives | SGNet · PSI trajectory prediction, 1.5s | ADE: 57.81 to 53.71 | Helped |
Surprising findings
- Pedestrian intent and driving decisions predicted by LMMs are often mutually inconsistent.
- DeepSeek R1 often failed to detect any pedestrians in the scene, producing estimations based on general knowledge.
Limitations
- Small dataset of 196 encounters, limited to naturalistic driving data from one study.
- Off-the-shelf VLMs struggle to perceive pedestrian direction and produce verbose, hallucinated reasoning.
Experimental setup
Datasets: PSI TASI 110-car naturalistic driving study
Benchmarks: PSI
Metrics: Accuracy F1 Acc_avg CIDEr BLEU-4 ROUGE METEOR BERTScore ADE FDE
Baselines: LSTM Transformer PIE BiTraP-D BiTraP-NP SGNet SGNet+evi Qwen-VL InternVL SmolVLM LLaMA 3.2 Vision Gemini 2.5 Pro
Builds on: PIE BiTraP SGNet
Closest papers by meaning
- DriveCombo: Benchmarking Compositional Traffic Rule Reasoning in Autonomous Driving (CVPR 2026 · similarity 0.87)
- BAT: Behavior-Aware Human-Like Trajectory Prediction for Autonomous Driving (AAAI 2024 · similarity 0.86)
- WOMD-Reasoning: A Large-Scale Dataset for Interaction Reasoning in Driving (ICML 2025 · similarity 0.86)
- Continuously Learning, Adapting, and Improving: A Dual-Process Approach to Autonomous Driving (NeurIPS 2024 · similarity 0.85)
- DriveGPT: Scaling Autoregressive Behavior Models for Driving (ICML 2025 · similarity 0.85)
- Dual-View Interaction-Aware Lane Change Prediction for Autonomous Driving (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.