Research map / Probabilistic and scientific modeling / Molecules and proteins
Leveraging Driver Field-of-View for Multimodal Ego-Trajectory Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RouteFormer, a multimodal ego-trajectory predictor fusing GPS history, surrounding-camera video, and driver field-of-view (headcam video plus gaze), along with the PCI complexity metric and the GEM dataset.
Problem
Predicting the ego-vehicle's future trajectory is under-explored compared with other-agent prediction, and existing methods ignore driver attention and intent; gaze-annotated driving datasets are also scarce.
Approach
Fuse driver field-of-view (first-person video and gaze fixations) with scene video and past GPS through a time-series transformer, trained with an auxiliary future-visual-feature loss and a future-discounted loss; additionally define a Fréchet-distance-based Path Complexity Index to stratify scenario difficulty.
Key results
- On GEM, RouteFormer reaches 5.99 m ADE versus 6.13 m for RouteFormer-Base and 6.07 m for Multimodal Transformer; on the 20+ PCI subset ADE is 7.70 m vs 8.14 m.
- On DR(eye)VE, RouteFormer achieves 8.75 m ADE vs 10.67 m for Multimodal Transformer and 9.95 m for RouteFormer-Base.
- Gains from driver FOV grow with prediction horizon, reaching 6.21% FDE improvement at 6 s on high-PCI samples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add driver FOV (video and gaze) to video-only RouteFormer (ablation) | RouteFormer · GEM test | ADE (m): 6.13 -> 5.99 | Helped |
| Add driver FOV (video and gaze) to video-only RouteFormer (ablation) | RouteFormer · DR(eye)VE test | ADE (m): 9.95 -> 8.75 | Helped |
| Add driver FOV (video and gaze) to video-only RouteFormer (ablation) | RouteFormer · GEM test | ADE+20PCI (m): 8.14 -> 7.70 | Helped |
| Add driver FOV (video and gaze) to video-only RouteFormer (ablation) | RouteFormer · DR(eye)VE test | ADE+20PCI (m): 13.15 -> 12.26 | Helped |
| Add auxiliary losses (ablation) | RouteFormer · GEM validation, 20+ PCI | Val. ADE (m): 8.85 -> 8.68 | Helped |
| Add future-discounted loss after adding auxiliary losses (ablation) | RouteFormer · GEM validation, 20+ PCI | Val. ADE (m): 8.68 -> 8.27 | Helped |
| Replace the time-series module with Informer rather than Transformer (ablation) | RouteFormer · GEM validation, 20+ PCI | Val. ADE (m): 9.97 -> 9.62 | Helped |
| Use SwinV2 instead of Informer without a pretrained visual encoder (ablation) | RouteFormer · GEM validation, 20+ PCI | Val. ADE (m): 9.62 -> 8.51 | Helped |
| Use SwinV2 instead of SAM as the visual encoder (ablation) | RouteFormer · GEM validation, 20+ PCI | Val. ADE (m): 9.08 -> 8.51 | Helped |
| Use SwinV2 instead of DinoV2 as the visual encoder (ablation) | RouteFormer · GEM validation, 20+ PCI | Val. ADE (m): 8.69 -> 8.51 | Helped |
| Remove the scene branch (ablation) | RouteFormer · GEM test | ADE (m): 5.99 -> 7.83 | Hurt |
| Remove both scene and FOV branches (ablation) | RouteFormer · GEM test | ADE (m): 5.99 -> 8.19 | Hurt |
Surprising findings
- Autobots, a state-of-the-art multi-agent predictor, performed much worse (10.87 m ADE on GEM) when adapted to ego-trajectory prediction.
- Adding gaze to a video-only model yields only a small improvement on the full test set (about 2%), with larger gains concentrated on complex high-PCI paths.
Limitations
- GPS ground truth required manual correction, and the dataset is small (5 hours, 10 participants).
- Failure cases occur in right turns occluded by trees or parked cars and at complex intersections with multiple path options.
- Gaze and scene video registration could not be achieved reliably via stitching, so modalities are kept as separate inputs.
Experimental setup
Datasets: GEM DR(eye)VE
Benchmarks: GEM DR(eye)VE
Metrics: ADE FDE ADE+20PCI
Baselines: Stationary baseline Linear baseline GIMO Multimodal Transformer RouteFormer-Base Autobots MTR
Builds on: Informer GIMO SwinV2 nuScenes task setting
Closest papers by meaning
- FlowAD: Ego-Scene Interactive Modeling for Autonomous Driving (ICLR 2026 · similarity 0.88)
- DriverGaze360: OmniDirectional Driver Attention with Object-Level Guidance (CVPR 2026 · similarity 0.88)
- BAT: Behavior-Aware Human-Like Trajectory Prediction for Autonomous Driving (AAAI 2024 · similarity 0.86)
- On-Road Object Importance Estimation: A New Dataset and A Model with Multi-Fold Top-Down Guidance (NeurIPS 2024 · similarity 0.85)
- The Road Ahead in Autonomous Driving: The KITScenes Multimodal Dataset (NeurIPS 2026 · similarity 0.85)
- Prioritizing Perception-Guided Self-Supervision: A New Paradigm for Causal Modeling in End-to-End Autonomous Driving (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.