Research map / Vision-language and multimodal / Cross-modal learning
Multi-modal Knowledge Distillation-based Human Trajectory Forecasting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills a full-modality teacher (trajectory, 3D human pose, text) into trajectory-only or trajectory+pose student trajectory forecasters for ego-view and BEV pedestrian datasets, using regression plus latent KL/cosine KD losses.
Problem
Pedestrian trajectory forecasting benefits from extra modalities such as human pose and text, but online text extraction needs a VLM that is too costly for resource-constrained mobile systems.
Approach
Train a teacher with trajectory, SMPL pose, and text, then distill its intra-agent fusion latent (Q) and inter-agent interaction latent (H) into a student using only trajectory or trajectory+pose, across full and instantaneous (1–2 frame) observations.
Key results
- On SIT, the HiVT X+P student with KD improves by about 13% (13.03) average over the no-KD baseline.
- On JRDB, MART X student with KD improves by 7.61% average; HiVT X student improves by 2.38%.
- Teacher with X+P+S improves over X alone by up to 14.98% (MART on JRDB) and up to 27% on ADE1 for MART in instantaneous settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add text to trajectory and pose inputs | JRDB HiVT teacher · JRDB | ADE1: 0.364 -> 0.280 | Helped |
| Add text to trajectory and pose inputs | JRDB MART teacher · JRDB | ADE1: 0.366 -> 0.289 | Helped |
| Distill a trajectory-only student from the full-modality teacher | HiVT · SIT | ADE1: 0.556 -> 0.503 | Helped |
| Distill a trajectory-and-pose student from the full-modality teacher | HiVT · SIT | ADE: 0.518 -> 0.414 | Helped |
| Add intra-agent distillation (ablation) | HiVT X + P · JRDB | ADE1: 0.659 -> 0.647 | Helped |
| Add inter-agent distillation (ablation) | HiVT X + P · JRDB | ADE1: 0.659 -> 0.601 | Helped |
| Add both intra-agent and inter-agent distillation (ablation) | HiVT X + P · JRDB | ADE1: 0.659 -> 0.560 | Helped |
| Add intra-agent distillation (ablation) | MART X + P · JRDB | ADE: 0.287 -> 0.274 | Helped |
| Add inter-agent distillation (ablation) | MART X + P · JRDB | ADE: 0.287 -> 0.275 | Helped |
| Add interaction text (ablation) | HiVT teacher using X + P + S · JRDB | ADE: 0.233 -> 0.225 | Helped |
| Use obstacle-context captions rather than person-action captions (ablation) | HiVT X + S teacher · SIT | ADE: 0.521 -> 0.506 | Helped |
| Use future-behavior captions rather than person-action captions (ablation) | HiVT X + S teacher · SIT | ADE: 0.521 -> 0.520 | No clear effect |
Surprising findings
- Adding 3D human pose alone (X+P) degrades HiVT performance on JRDB (-2.84%), while adding text (X+S) helps substantially.
- Text captions bridge the domain gap between trajectory and noisy pose, so text is key to integrating pose.
Limitations
- Text for SIT is generated by a VLM (PLLaVA) and the ETH/UCY text is rule-based, not VLM-derived.
- Pose extraction noise limits the benefit of pose alone.
- Gains on ETH/UCY are modest (about 1-4%).
Experimental setup
Datasets: JRDB SIT ETH/UCY
Benchmarks: JRDB SIT ETH/UCY
Metrics: ADE FDE ADE1 ADE2 FDE1 FDE2
Baselines: HiVT MART ST (SocialTransmotion) LED
Builds on: HiVT MART LED SocialTransmotion
Closest papers by meaning
- Social-Transmotion: Promptable Human Trajectory Prediction (ICLR 2024 · similarity 0.87)
- AutoTraces: Autoregressive Trajectory Forecasting via Multimodal Large Language Models (CVPR 2026 · similarity 0.87)
- Multi-agent Long-term 3D Human Pose Forecasting via Interaction-aware Trajectory Conditioning (CVPR 2024 · similarity 0.87)
- Multimodal Sense-Informed Forecasting of 3D Human Motions (CVPR 2024 · similarity 0.86)
- MoFlow: One-Step Flow Matching for Human Trajectory Forecasting via Implicit Maximum Likelihood Estimation based Distillation (CVPR 2025 · similarity 0.86)
- Towards Predicting Any Human Trajectory In Context (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.