Research map / 3D vision / 3D generation
SEPT: Towards Efficient Scene Representation Learning for Motion Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SEPT, a transformer motion predictor pretrained with three masked-reconstruction self-supervised tasks (trajectory, road, and tail prediction) that reaches state-of-the-art on Argoverse 1 and 2 forecasting.
Problem
Motion forecasting for autonomous vehicles needs effective spatiotemporal scene encoding; prior SOTA relies on sophisticated architectures and feature engineering, and existing SSL for motion prediction is less effective.
Approach
A compact sequential temporal-then-spatial transformer encoder pretrained with three self-supervised tasks (masked trajectory modeling, attribute-level masked road modeling, and tail prediction that jointly uses trajectory and road context) before end-to-end finetuning.
Key results
- On Argoverse 1 test leaderboard, SEPT achieves b-minFDE6 1.682, minADE6 0.728, minFDE6 1.057, ranking first on all metrics except MR6 (where HOME is best).
- On Argoverse 2 leaderboard, SEPT achieves b-minFDE6 1.736, minADE6 0.605, minFDE6 1.151, MR6 0.137, ranking first on all metrics versus QCNet's b-minFDE6 1.779.
- SEPT uses about 40% of QCNet's parameters (~9.6M) with roughly twice faster inference; ablations show the three pretraining tasks improve performance additively.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SEPT instead of QCNet | SEPT · Argoverse 1 leaderboard | b-minFDE6: 1.693 -> 1.682 | Helped |
| Use SEPT instead of QCNet | SEPT · Argoverse 2 leaderboard | b-minFDE6: 1.779 -> 1.736 | Helped |
| Add the Tail Prediction (TP) pretraining task (ablation) | SEPT scene understanding pretraining · Argoverse 1 validation | b-minFDE6 | Helped |
| Add Masked Trajectory Modeling (MTM) pretraining (ablation) | SEPT scene understanding pretraining · Argoverse 1 validation | b-minFDE6 | Helped |
| Add Masked Road Modeling (MRM) pretraining (ablation) | SEPT scene understanding pretraining · Argoverse 1 validation | b-minFDE6 | Helped |
| Vary the MTM or MRM mask ratio within suitable ranges (ablation) | SEPT with the corresponding single pretraining task · Argoverse 1 validation | b-minFDE6 | No clear effect |
| Vary visible-history length T_h within suitable ranges (ablation) | SEPT with TP pretraining · Argoverse 1 validation | b-minFDE6 | No clear effect |
| Add the validation split to pretraining data (ablation) | SEPT · Argoverse 1 validation | b-minFDE6: 1.577 -> 1.569 | Helped |
| Add the test split to pretraining data (ablation) | SEPT · Argoverse 1 validation | b-minFDE6: 1.569 -> 1.563 | Helped |
Surprising findings
- Forecast-MAE ablations reportedly show some task combinations degrade performance versus training from scratch, whereas SEPT's combinations consistently help.
Limitations
- Single-agent, agent-centric prediction; not trivially extendable to joint multi-agent prediction.
- Miss rate MR6 on Argoverse 1 is not best (HOME leads via sparse sampling).
Experimental setup
Datasets: Argoverse 1 Argoverse 2
Benchmarks: Argoverse 1 leaderboard Argoverse 2 leaderboard
Metrics: b-minFDE6 minADE6 minFDE6 MR6 minADE1 minFDE1 MR1
Baselines: DenseTNT HOME MultiPath++ GANet MacFormer DCMS Gnet Wayformer ProphNet QCNet Traj-MAE Forecast-MAE
Builds on: Masked autoencoder VectorNet T5 relative position embedding Traj-MAE Forecast-MAE
Closest papers by meaning
- SmartPretrain: Model-Agnostic and Dataset-Agnostic Representation Learning for Motion Prediction (ICLR 2025 · similarity 0.89)
- Enhancing End-to-End Autonomous Driving with Latent World Model (ICLR 2025 · similarity 0.87)
- Motion Forecasting in Continuous Driving (NeurIPS 2024 · similarity 0.87)
- Visual Point Cloud Forecasting enables Scalable Autonomous Driving (CVPR 2024 · similarity 0.86)
- S4-Driver: Scalable Self-Supervised Driving Multimodal Large Language Model with Spatio-Temporal Visual Representation (CVPR 2025 · similarity 0.86)
- HPNet: Dynamic Trajectory Forecasting with Historical Prediction Attention (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.