Research map / Label-efficient and robust learning / Test-time adaptation
T4P: Test-Time Training of Trajectory Prediction via Masked Autoencoder and Actor-specific Token Memory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes T4P, a test-time training method for trajectory prediction that adds a masked-autoencoder reconstruction loss to delayed-ground-truth regression and keeps an actor-specific token memory, enabling updates of deeper network layers under cross-dataset shifts.
Problem
Trajectory prediction models become unreliable under distribution shifts at test time, and prior online methods only update the last decoder layer because of few delayed-label samples, which limits adaptation.
Approach
Combine masked-autoencoder reconstruction loss with delayed ground-truth regression loss so deeper encoder and embedding layers can be safely updated online, plus an actor-specific token memory that carries per-instance motion characteristics across time within a scene and across scenes via class-averaged tokens.
Key results
- On cross-dataset shifts in the short-term setting, T4P reaches mADE6 0.391 on INTER→Lyft versus 0.746 for the best baseline MEK (tau=tf).
- Mean mADE6 across short-term shifts is 0.421 for T4P versus 0.691 for MEK (tau=tf).
- T4P keeps real-time speed (up to 24.2 FPS at lower update frequency) versus MEK at 3.3 FPS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add masked-autoencoder reconstruction loss alongside regression loss (ablation) | T4P · INTER → Lyft (short-term) | mADE6: 0.674 -> 0.391 | Helped |
| Add masked-autoencoder reconstruction loss alongside regression loss (ablation) | T4P · nuScenes → Lyft (long-term) | mADE6: 0.942 -> 0.776 | Helped |
| Add actor reconstruction loss (ablation) | T4P · INTER → Lyft (short-term) | mADE6: 0.674 -> 0.407 | Helped |
| Add lane reconstruction loss (ablation) | T4P · INTER → Lyft (short-term) | mADE6: 0.407 -> 0.391 | Helped |
| Optimize encoder and decoder with regression loss only (ablation) | T4P · nuScenes → Lyft (long-term) | mADE6: 0.840 -> 0.942 | Hurt |
| Add reconstruction loss when optimizing encoder and decoder (ablation) | T4P · nuScenes → Lyft (long-term) | mADE6: 0.942 -> 0.776 | Helped |
| Add actor-specific token memory (ablation) | T4P · INTER → Lyft (short-term) | mADE6: 0.581 -> 0.391 | Helped |
| Add actor-specific token memory (ablation) | T4P · nuScenes → Lyft (long-term) | mADE6: 0.932 -> 0.776 | Helped |
| Vary actor and lane masking ratios (ablation) | T4P · INTER → Lyft (short-term) | mADE6 | Mixed |
| Use T4P instead of MEK (τ = tf) | ForecastMAE · INTER → nuScenes (short-term) | mADE6: 0.892 -> 0.537 | Helped |
| Use T4P instead of MEK (τ = tf) | ForecastMAE · INTER → Lyft (short-term) | mADE6: 0.746 -> 0.391 | Helped |
| Use T4P instead of MEK (τ = tf) | ForecastMAE · nuScenes → Lyft (long-term) | mADE6: 1.006 -> 0.776 | Helped |
Surprising findings
- Regression-only loss degrades performance when updating embedding layers, while adding the reconstruction loss makes deeper-layer updates beneficial.
- Dataset-averaged DUA and TENT-with-supervision gave little or negative gains over source-only.
Limitations
- Actor-specific token benefit depends on long scene lengths; shorter scenes reduce effectiveness.
- Masking-ratio sensitivity: performance follows a U-shape and depends on chosen ratios.
Experimental setup
Datasets: nuScenes Lyft Waymo Open Motion Dataset INTERACTION
Benchmarks: nuScenes Lyft Waymo Open Motion Dataset INTERACTION
Metrics: mADE6 mFDE6 FPS
Baselines: Source Only Joint Training DUA TENT (w/ sup) MEK AML
Builds on: ForecastMAE TTT-MAE MEK TENT DUA
Closest papers by meaning
- Enhancing Trajectory Prediction through Self-Supervised Waypoint Distortion Prediction (ICML 2024 · similarity 0.86)
- Towards Generalizable Trajectory Prediction using Dual-Level Representation Learning and Adaptive Prompting (CVPR 2025 · similarity 0.85)
- LaKD: Length-agnostic Knowledge Distillation for Trajectory Prediction with Any Length Observations (NeurIPS 2024 · similarity 0.85)
- Trajeglish: Traffic Modeling as Next-Token Prediction (ICLR 2024 · similarity 0.85)
- Social-Transmotion: Promptable Human Trajectory Prediction (ICLR 2024 · similarity 0.85)
- Tra-MoE: Learning Trajectory Prediction Model from Multiple Domains for Adaptive Policy Conditioning (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.