Research map / Probabilistic and scientific modeling / Molecules and proteins
SmartPretrain: Model-Agnostic and Dataset-Agnostic Representation Learning for Motion Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SmartPretrain, a model-agnostic and dataset-agnostic self-supervised pre-training framework for motion prediction combining trajectory contrastive learning and trajectory reconstruction, pre-trained on mixed driving datasets and applied to HiVT, HPNet, QCNet, and Forecast-MAE.
Problem
Motion prediction models for autonomous driving operate in a small-data regime, and existing self-supervised pre-training methods are tied to specific architectures and single datasets, limiting scalability and generalization.
Approach
Pre-train any motion prediction encoder using only trajectory-level pretext tasks (contrastive learning across agents and time windows with a momentum branch, plus reconstruction of another sub-scenario's trajectories), so it needs no map or architecture-specific embeddings, and sample scenarios from mixed datasets after standardizing their formats.
Key results
- SmartPretrain with data-scaled pre-training reduces QCNet minFDE, minADE and MR on Argoverse 2 validation by 4.9%, 3.3% and 7.6%.
- Forecast-MAE with SmartPretrain on Argoverse 2 validation reduces MR by 10.6% versus 6.2% for its own pre-training (Forecast-MAE pre-training).
- On HiVT (Argoverse), combining contrastive and reconstruction pretext tasks reduces minFDE by 3.0% on validation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SmartPretrain to HPNet using Argoverse pre-training | HPNet · Argoverse val | minFDE: 0.871 -> 0.854 | Helped |
| Add SmartPretrain to QCNet using all datasets | QCNet · Argoverse 2 val | MR: 0.157 -> 0.145 | Helped |
| Replace Forecast-MAE pre-training with SmartPretrain | Forecast-MAE · Argoverse 2 val | minFDE: 1.409 -> 1.372 | Helped |
| Pre-train HiVT on multiple datasets instead of Argoverse only (ablation) | HiVT · Argoverse val | minFDE: 0.940 -> 0.929 | Helped |
| Pre-train HiVT on a different dataset rather than the downstream dataset (ablation) | HiVT · Argoverse val | minFDE: 0.969 -> 0.951 | Mixed |
| Add trajectory contrastive learning (ablation) | HiVT · Argoverse val | minFDE: 0.969 -> 0.958 | Helped |
| Add trajectory reconstruction learning (ablation) | HiVT · Argoverse val | minFDE: 0.969 -> 0.952 | Helped |
| Combine contrastive and reconstructive pretext tasks (ablation) | HiVT · Argoverse val | minFDE: 0.969 -> 0.940 | Helped |
| Use a predictive reconstruction target instead of historical input trajectory (ablation) | HiVT · Argoverse val | minFDE: 0.956 -> 0.940 | Helped |
| Use standard motion prediction rather than SSL to pre-train on additional datasets (ablation) | HiVT · Argoverse val | minFDE: 0.969 -> 1.077 | Hurt |
Surprising findings
- Naively pre-training on the standard motion prediction task with additional datasets (Argoverse 2) or directly mixing datasets hurts performance versus training on Argoverse alone (appendix).
Limitations
- Only pre-trained HiVT and QCNet on both settings; HPNet and Forecast-MAE only on single dataset due to compute constraints.
- Gains are modest on some metrics (e.g., HPNet test minFDE -0.8%).
Experimental setup
Datasets: Argoverse Argoverse 2 Waymo Open Motion Dataset
Benchmarks: Argoverse validation Argoverse test Argoverse 2 validation Argoverse 2 test
Metrics: minFDE minADE MR
Baselines: HiVT HPNet QCNet Forecast-MAE PreTraM
Builds on: Forecast-MAE BYOL-style self-training MAE
Closest papers by meaning
- SEPT: Towards Efficient Scene Representation Learning for Motion Prediction (ICLR 2024 · similarity 0.89)
- SMART: Scalable Multi-agent Real-time Motion Generation via Next-token Prediction (NeurIPS 2024 · similarity 0.89)
- Semi-supervised Class-Agnostic Motion Prediction with Pseudo Label Regeneration and BEVMix (AAAI 2024 · similarity 0.88)
- SmartRefine: A Scenario-Adaptive Refinement Framework for Efficient Motion Prediction (CVPR 2024 · similarity 0.88)
- S4-Driver: Scalable Self-Supervised Driving Multimodal Large Language Model with Spatio-Temporal Visual Representation (CVPR 2025 · similarity 0.88)
- Motion Forecasting in Continuous Driving (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.