Research map / Probabilistic and scientific modeling / Molecules and proteins
ModeSeq: Taming Sparse Multimodal Motion Prediction with Sequential Mode Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ModeSeq, which decodes trajectory modes sequentially (one per step) with an Early-Match-Take-All loss and iterative mode rearrangement, achieving sparse yet diverse multimodal motion prediction on Waymo and Argoverse 2.
Problem
Multimodal motion prediction for traffic agents suffers from lack of multimodal ground truth, causing winner-take-all training to yield limited trajectory diversity and miscalibrated confidence; dense mode prediction with NMS post-processing is a heuristic workaround that hurts accuracy.
Approach
Model the K future modes as a sequence: a transformer decoder produces one mode embedding per step conditioned on previously decoded modes (memory transformer + context transformer), trained with an Early-Match-Take-All loss that assigns the positive label to the earliest matching mode, plus mode rearrangement between stacked layers, enabling sparse outputs and mode extrapolation at test time.
Key results
- On the WOMD validation set, ModeSeq reaches 0.4507 mAP6 and 0.1206 MR6 versus QCNet's 0.4452 and 0.1254, with slightly worse minADE6 (0.5237 vs 0.5122).
- On the Argoverse 2 benchmark, ensemble-free ModeSeq achieves b-minFDE6 1.87, MR6 0.14, minADE6 0.63, minFDE6 1.26, surpassing QCNet (1.91, 0.16, 0.65, 1.29).
- ModeSeq ranked first among Lidar-free approaches on the WOMD 2024 test set in its ensemble version (mAP6 0.4665).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace parallel DETR-style decoding with sequential mode decoding (ablation) | motion prediction on WOMD · WOMD validation, 20% training data | Soft mAP6: 0.4050 -> 0.4138 | Helped |
| use EMTA instead of WTA (ablation) | ModeSeq on WOMD · WOMD validation, 20% training data | Soft mAP6: 0.4138 -> 0.4231 | Helped |
| use EMTA instead of WTA (ablation) | ModeSeq on WOMD · WOMD validation, 20% training data | minADE6: 0.5563 -> 0.5700 | Hurt |
| ignore confidence loss for other matched modes under WTA (ablation) | ModeSeq on WOMD · WOMD validation, 20% training data | Soft mAP6: 0.4138 -> 0.4207 | Helped |
| ignore confidence loss for other matched modes under EMTA (ablation) | ModeSeq on WOMD · WOMD validation, 20% training data | Soft mAP6: 0.4231 -> 0.4098 | Hurt |
| add mode rearrangement between decoding layers (ablation) | ModeSeq on WOMD · WOMD validation, 20% training data | Soft mAP6: 0.4112 -> 0.4231 | Helped |
| ignore confidence loss for early mismatches without mode rearrangement (ablation) | ModeSeq on WOMD · WOMD validation, 20% training data | Soft mAP6: 0.4112 -> 0.4141 | Helped |
| ignore confidence loss for early mismatches with mode rearrangement (ablation) | ModeSeq on WOMD · WOMD validation, 20% training data | Soft mAP6: 0.4231 -> 0.4161 | Hurt |
| use ModeSeq instead of QCNet | motion prediction · WOMD 2024 validation | Soft mAP6: 0.4508 -> 0.4562 | Helped |
| use ModeSeq instead of QCNet | motion prediction · WOMD 2024 validation | minADE6: 0.5122 -> 0.5237 | Hurt |
| use ModeSeq instead of QCNet | motion prediction · Argoverse 2 validation | minADE6: 0.65 -> 0.63 | Helped |
| use ModeSeq instead of QCNet | motion prediction · Argoverse 2 validation | MR6: 0.16 -> 0.14 | Helped |
Surprising findings
- Treating other matched modes as ignored samples helps under WTA but is detrimental under EMTA training.
- A 3-mode ModeSeq achieves mAP6 on par with the 6-mode model and better than 6-mode QCNet.
Limitations
- Inference latency roughly doubles QCNet's when predicting 6 modes.
- Slight degradation in minADE6 and minFDE6 relative to QCNet.
- Lags behind MTR v3 on mAP6 because MTR v3 uses lidar input.
Experimental setup
Datasets: Waymo Open Motion Dataset Argoverse 2 Motion Forecasting Dataset
Benchmarks: Waymo Open Motion Dataset motion prediction benchmark Argoverse 2 Single-Agent Motion Forecasting Benchmark
Metrics: mAP6 Soft mAP6 MR6 minADE6 minFDE6 b-minFDE6
Baselines: QCNet MTR v3 MTR++ RMP Ensemble MTR DETR-like decoder with refinement
Builds on: QCNet DETR winner-take-all MTR
Closest papers by meaning
- DeMo: Decoupling Motion Forecasting into Directional Intentions and Dynamic States (NeurIPS 2024 · similarity 0.88)
- UniMotion: A Unified Motion Framework for Simulation, Prediction and Planning (NeurIPS 2025 · similarity 0.87)
- EDA: Evolving and Distinct Anchors for Multimodal Motion Prediction (AAAI 2024 · similarity 0.86)
- Trajeglish: Traffic Modeling as Next-Token Prediction (ICLR 2024 · similarity 0.86)
- Generating Multimodal Driving Scenes via Next-Scene Prediction (CVPR 2025 · similarity 0.86)
- BehaviorGPT: Smart Agent Simulation for Autonomous Driving with Next-Patch Prediction (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.