Research map / Vision-language and multimodal / Multimodal LLMs

AutoTraces: Autoregressive Trajectory Forecasting via Multimodal Large Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AutoTraces, an autoregressive vision-language-trajectory model built on LLaVA-NeXT-Video-7B that tokenizes waypoints as special point tokens with learned point embeddings and generates socially compliant robot trajectories, with automated CoT from Qwen-VL-Max.

Problem

Socially compliant robot trajectory forecasting in human-populated environments generalizes poorly, and text-coordinate LLM methods are token-inefficient while non-autoregressive methods fix output length and struggle with long horizons and flexible-length forecasting.

Approach

A point token plus a learned point encoder/decoder (Point Encoder and Point Head) inserts waypoints into a multimodal LLM's native autoregressive generation, combined with automatically generated CoT rationales from a VLM using trajectory curvature/meta-action analysis, trained in two stages with QLoRA.

Key results

• On SCAND at T=10, AutoTraces achieves L2 1.089m and L1 1.384m, beating the best baseline CityWalker (1.407m L2) by 0.318m.
• On cross-scene RECON at T=10, AutoTraces reduces L2 error by 32.6% relative to LLaVa-Video.
• For horizons 12-20 on SCAND, AutoTraces reaches 99.92% instruction execution accuracy vs 40.34% for LLaVa-Video, with 25 tokens per response vs 375.

Surprising findings

• At T=5 on SCAND, the single-model GNM variant trained on fixed length and the unified baselines perform comparably, and a fixed-length single-pass variant of AutoTraces is only slightly worse on SCAND, suggesting the autoregressive gain shows mainly on unseen RECON.

Limitations

• Relies on closed Qwen-VL-Max for CoT generation, which is costly and not reproducible.
• Evaluated mainly on SCAND with cross-scene tests on GoStanford and RECON; limited to 2D waypoints and short horizons with 1 Hz sampling.
• Some baseline comparisons use a unified model variant and non-standard truncated-length evaluation, complicating fair comparison.

Experimental setup

Datasets: SCAND GoStanford RECON

Benchmarks: SCAND GoStanford RECON

Metrics: L2 distance (m) L1 distance (m) IEAcc TPR

Baselines: GNM ViNT NoMad CityWalker LLaVa-Video

Builds on: LLaVA-Video LLaVA-NeXT-Video QLoRA ViNT CityWalker

Closest papers by meaning

• MolmoMotion: Forecasting Point Trajectories in 3D with Language Instruction (NeurIPS 2026 · similarity 0.88)
• Multi-modal Knowledge Distillation-based Human Trajectory Forecasting (CVPR 2025 · similarity 0.87)
• Can Language Beat Numerical Regression? Language-Based Multimodal Trajectory Prediction (CVPR 2024 · similarity 0.87)
• ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model (NeurIPS 2026 · similarity 0.87)
• SAMPO: Scale-wise Autoregression with Motion Prompt for Generative World Models (NeurIPS 2025 · similarity 0.86)
• SmartPretrain: Model-Agnostic and Dataset-Agnostic Representation Learning for Motion Prediction (ICLR 2025 · similarity 0.86)