Research map / 3D vision / Human pose estimation
Social-Transmotion: Promptable Human Trajectory Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Social-Transmotion, a dual-Transformer (CMT then ST) model that treats trajectories, 2D/3D poses and bounding boxes as promptable cues with modality and meta masking for human trajectory prediction.
Problem
Human trajectory prediction usually uses only x-y ground coordinates, ignoring non-verbal visual cues such as body pose and bounding boxes; existing models also cannot flexibly handle varying or missing cue sets.
Approach
Translate the NLP prompt idea to trajectory prediction: a cross-modality Transformer encodes each agent's trajectory plus any available cues (2D/3D pose, bounding boxes) into motion tokens, then a Social Transformer models inter-agent interactions; modality-masking and meta-masking during training let one model handle any cue subset at inference.
Key results
- On JTA, Social-Transmotion with trajectory only reaches ADE/FDE 0.99/1.98, beating EqMotion's 1.13/2.39; adding 3D pose gives 0.89/1.81 and all cues give 0.88/1.80.
- On JRDB, trajectory plus 2D bounding boxes gives ADE/FDE 0.37/0.73, the best reported result versus EqMotion 0.42/0.78.
- On ETH-UCY, the model achieves average ADE/FDE 0.51/1.03, competitive with EqMotion 0.49/1.03 (trajectory-only).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add 3D pose to trajectories | Social-Transmotion · JTA | ADE: 0.99 -> 0.89 | Helped |
| Add 2D pose to trajectories | Social-Transmotion · JTA | ADE: 0.99 -> 0.95 | Helped |
| Add 2D bounding boxes to trajectories | Social-Transmotion · JRDB | ADE: 0.40 -> 0.37 | Helped |
| Add 3D bounding boxes to trajectories and 3D pose | Social-Transmotion · JTA | ADE: 0.89 -> 0.89 | No clear effect |
| Use all available visual cues with trajectories | Social-Transmotion · JTA | ADE: 0.89 -> 0.88 | Helped |
| Train once on all modalities with masking instead of training a specific model | Social-Transmotion · JTA, T + 3D P | ADE: 0.92 -> 0.89 | Helped |
| Use pose only for the primary agent rather than all agents (ablation) | Social-Transmotion · JTA | ADE: 0.89 -> 0.92 | Hurt |
| Use only the last observed pose (ablation) | Social-Transmotion · JTA | ADE: 0.89 -> 0.94 | Hurt |
| Use only head keypoints (ablation) | Social-Transmotion · JTA | ADE: 0.89 -> 0.98 | Hurt |
| Replace CMT with an MLP (ablation) | CMT-ST · JTA | ADE: 0.92 -> 1.06 | Hurt |
| Reverse the order of CMT and ST (ablation) | CMT-ST · JTA | ADE: 0.92 -> 1.13 | Hurt |
| Remove ST (ablation) | CMT-ST · JTA | ADE: 0.92 -> 1.23 | Hurt |
Surprising findings
- Adding visual cues did not consistently help; the appendix notes instances where they failed to improve performance.
- Using only the last observed pose gives performance comparable to using all time-steps, and head-only pose gives little gain over trajectory alone.
Limitations
- Only a limited set of visual cues (poses, bounding boxes) was examined; gaze, actions and scene context were not explored.
- Cues do not always improve performance; failure cases involve crowds, abrupt path changes and missing scene context.
- Focuses only on deterministic prediction with a single trajectory per agent.
Experimental setup
Datasets: JTA JRDB Pedestrians and Cyclists in Road Traffic ETH-UCY
Benchmarks: JTA JRDB Pedestrians and Cyclists in Road Traffic ETH-UCY
Metrics: ADE FDE ASWAEE
Baselines: Social-GAN-det Transformer Vanilla-LSTM Occupancy-LSTM Directional-LSTM Dir-social-LSTM Social-LSTM Autobots Trajectron++ EqMotion MemoNet
Builds on: Transformer TrajNet++ Social-GAN EqMotion
Closest papers by meaning
- SocialCVAE: Predicting Pedestrian Trajectory via Interaction Conditioned Latents (AAAI 2024 · similarity 0.90)
- SimpliHuMoN: Simplifying Human Motion Prediction (NeurIPS 2026 · similarity 0.88)
- SocialMOIF: Multi-Order Intention Fusion for Pedestrian Trajectory Prediction (CVPR 2025 · similarity 0.88)
- SocialCircle: Learning the Angle-based Social Interaction Representation for Pedestrian Trajectory Prediction (CVPR 2024 · similarity 0.88)
- Multi-modal Knowledge Distillation-based Human Trajectory Forecasting (CVPR 2025 · similarity 0.87)
- Following the Human Thread in Social Navigation (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.