Research map / Vision-language and multimodal / Visual question answering

Can Language Beat Numerical Regression? Language-Based Multimodal Trajectory Prediction

CVPR 2024 · Vision-language and multimodal › Visual question answering · approach: Multimodal LLM question answering · 47 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Recasts pedestrian trajectory prediction as a question-answering task over text prompts, using GPT-3.5/GPT-4 zero-shot (LMTraj-ZERO) and a fine-tuned T5 with a numeric BPE tokenizer and multi-task social QA (LMTraj-SUP).

Problem

Pedestrian trajectory prediction is dominated by numerical regression models that treat coordinates as continuous signals and capture social interaction poorly; language models' reasoning ability has not been exploited for this numerical forecasting task.

Approach

Convert trajectories into decimal-text prompts, image scenes into BLIP-2 captions, and wrap them in QA templates; use a numeric-optimized tokenizer, auxiliary social-relation QA tasks, and beam search / temperature sampling to produce deterministic and multimodal forecasts from a language model.

Key results

• LMTraj-SUP reaches average deterministic ADE/FDE of 0.48/0.88 m on ETH/UCY versus 0.51/1.11 for the prior best deterministic baseline in Table 4.
• LMTraj-ZERO with GPT-4 achieves average ADE/FDE of 0.39/0.79 m on ETH/UCY in zero-shot setting, beating linear and Kalman baselines.
• Numerical tokenizer lowers average ETH/UCY deterministic ADE/FDE from 0.65/1.28 (pretrained) to 0.48/0.88.

Measured effects reported in the paper

Surprising findings

• Zero-shot GPT-4 prompting performs comparably to the supervised Social-STGCNN baseline on ETH/UCY.

Limitations

• Zero-shot GPT-based inference requires about 20 seconds per API call, making it slow and costly.
• Autoregressive token generation is slower than the fastest numerical models, though still real-time.

Experimental setup

Datasets: ETH UCY SDD GCS

Benchmarks: ETH UCY SDD GCS

Metrics: ADE FDE ROUGE-1

Baselines: Linear Kalman filter AutoTrajectory Social-LSTM Social-GAN STGAT PECNet MID AgentFormer SocialVAE EigenTrajectory Social-STGCNN

Builds on: T5 BPE BLIP-2 AgentFormer Social-LSTM

Closest papers by meaning

• AutoTraces: Autoregressive Trajectory Forecasting via Multimodal Large Language Models (CVPR 2026 · similarity 0.87)
• Multi-modal Knowledge Distillation-based Human Trajectory Forecasting (CVPR 2025 · similarity 0.85)
• RHYTHM: Reasoning with Hierarchical Temporal Tokenization for Human Mobility (NeurIPS 2025 · similarity 0.85)
• Social-Transmotion: Promptable Human Trajectory Prediction (ICLR 2024 · similarity 0.85)
• GigaTraj: Predicting Long-term Trajectories of Hundreds of Pedestrians in Gigapixel Complex Scenes (CVPR 2024 · similarity 0.84)
• The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion (CVPR 2025 · similarity 0.84)