Research map / Vision-language and multimodal / Human and embodied modeling
MoST: Multi-Modality Scene Tokenization for Motion Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoST, which tokenizes multi-frame LiDAR and camera scenes into open-vocabulary scene elements encoded by pre-trained image models and point features, fused into a Wayformer motion predictor, and releases camera embeddings for WOMD.
Problem
Motion prediction models rely on symbolic perception outputs (boxes, road graph, traffic lights), which are vulnerable to perception errors and miss scene context; end-to-end sensor learning is costly and less interpretable.
Approach
Decompose the LiDAR scene into ground tiles, agent boxes and open-set object clusters, pool per-point features from pre-trained 2D image encoders (SAM ViT-H, CLIP, DINOv2, VQ-GAN) plus point and box geometry, and fuse them over time into a few hundred scene tokens fed alongside symbolic inputs to a transformer motion predictor.
Key results
- MoST-SAM H-64 reaches soft-mAP 0.4380 and minADE 0.5487 on WOMD validation vs. reproduced Wayformer 0.4239 soft-mAP with 3 decoders.
- Relative gains reported as 6.6% soft-mAP or 10.3% minADE improvement over the baseline in the abstract/intro.
- Under 50% simulated perception failure, MoST performs on par with the Wayformer baseline (soft-mAP 0.4164 vs 0.4121 cited).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add multi-modality scene tokens using SAM ViT-H features and 64 queries | Wayformer · WOMD validation | soft-mAP: 0.4239 -> 0.4380 | Helped |
| Add multi-modality scene tokens using SAM ViT-H features and 64 queries | Wayformer · WOMD validation | minADE: 0.5494 -> 0.5487 | Helped |
| Use SAM ViT-H image features instead of other tested image encoders (ablation) | Single-frame MoST · WOMD validation | minADE: 0.5670 -> 0.5483 | Helped |
| Add image features (ablation) | Wayformer · WOMD validation | minADE: 0.5830 -> 0.5483 | Helped |
| Add LiDAR features to camera image features (ablation) | Single-frame MoST · WOMD validation | mAP: 0.4118 -> 0.4162 | Helped |
| Add open-set elements (ablation) | Single-frame MoST · WOMD validation | soft-mAP: 0.4112 -> 0.4321 | Helped |
| Add agent elements (ablation) | Single-frame MoST · WOMD validation | soft-mAP: 0.4273 -> 0.4321 | Helped |
| Add ground elements (ablation) | Single-frame MoST · WOMD validation | soft-mAP: 0.4241 -> 0.4321 | Helped |
| Aggregate scene elements across frames (ablation) | MoST · WOMD validation | soft-mAP: 0.4321 -> 0.4380 | Helped |
| Replace scene-cluster tokens with image-grid tokens (ablation) | Single-frame MoST · WOMD validation | mAP: 0.4162 -> 0.4109 | Hurt |
Surprising findings
- Image-grid tokens with 256 per camera give smaller gains than the cluster-based sparse scene tokens.
Limitations
- Raw camera images are not released, only pre-extracted tokens and embeddings.
- Evaluation limited to the validation set and marginal motion prediction on WOMD.
- Minimal improvement in minADE on hard scenarios remains a challenge.
Experimental setup
Datasets: Waymo Open Motion Dataset
Benchmarks: Waymo Open Motion Dataset validation set
Metrics: minADE minFDE miss rate mAP soft-mAP
Baselines: Wayformer MTR MotionLM MultiPath++ SceneTransformer MotionCNN
Builds on: Wayformer SAM WOMD
Closest papers by meaning
- SEPT: Towards Efficient Scene Representation Learning for Motion Prediction (ICLR 2024 · similarity 0.86)
- Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understanding (NeurIPS 2024 · similarity 0.85)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.85)
- LASER: A Neuro-Symbolic Framework for Learning Spatio-Temporal Scene Graphs with Weak Supervision (ICLR 2025 · similarity 0.85)
- Advancing Semantic Future Prediction through Multimodal Visual Sequence Transformers (CVPR 2025 · similarity 0.85)
- Cross-Modal Distillation for 2D/3D Multi-Object Discovery from 2D Motion (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.