Research map / Probabilistic and scientific modeling / Molecules and proteins

Evaluation of Trajectory Distribution Predictions with Energy Score

ICML 2024 · Probabilistic and scientific modeling › Molecules and proteins · approach: Multimodal trajectory prediction

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that the Minimum-of-N family of trajectory prediction metrics (minADE/minFDE) is not strictly proper and proposes Energy Score-based evaluation (ES, EST, ESS) for trajectory distribution predictions, demonstrated on ETH/UCY with pre-trained models from Bae et al. (2022).

Problem

Common evaluation metrics for multimodal trajectory prediction, the "Minimum of N" family (minADE/minFDE), are not strictly proper scoring rules, so they can reward suboptimal distributional predictions and mis-rank models.

Approach

Analyze minADE/minFDE and their L-lowest generalizations within the proper scoring rules framework to prove impropriety and dependence on sample count K, then adopt the Energy Score (plus temporally/spatially marginalized variants EST and ESS) as a strictly proper, distribution-aware alternative.

Key results

• Theoretically proves ADE and ADE(L) for L<=2 are improper, and empirically shows minFDE/minADE minimized at higher-variance distributions while Energy Score is minimized at the true parameter as K grows.
• On ETH/UCY, Energy Score ranks SGCN-QMC above SGCN-NPSN on average, while minADE/minFDE prefer SGCN-NPSN consistently across datasets (e.g., SGCN-NPSN minADE AVG 0.36 vs SGCN-QMC 0.50).
• Table 1 shows ADE(L=1) decreases from 1.20 to 0.30 (x100) as K goes 20 to 100 for an optimal prediction, while Energy Score stays roughly stable (11.70 to 11.20).

Measured effects reported in the paper

Surprising findings

• minADE/minFDE can assign lower error to a suboptimal, lower-variance prediction than to the Bayes-optimal one (e.g., in a 1D Gaussian toy example).
• The NPSN sampling method, which optimizes minADE in its training objective, ranks best under minADE/minFDE but not under Energy Score.

Limitations

• Energy Score computation is O(K^2), though O(K) estimators exist and were not used.
• Variance of Energy Score estimates may be large for small K (K=10-20 used).
• Choice of metric is downstream-task dependent; the paper does not settle which model to choose across conflicting rankings.

Experimental setup

Datasets: ETH/UCY

Benchmarks: ETH/UCY

Metrics: Energy Score minADE minFDE ADE(L) FDE(L) EST ESS

Baselines: minADE minFDE ADE(L) FDE(L)

Builds on: Thiede & Brahma 2019 (MoN analysis) Gneiting & Raftery 2007 (proper scoring rules, energy score) Bae et al. 2022 (NPSN)

Closest papers by meaning

• Certified Human Trajectory Prediction (CVPR 2025 · similarity 0.85)
• LaKD: Length-agnostic Knowledge Distillation for Trajectory Prediction with Any Length Observations (NeurIPS 2024 · similarity 0.84)
• CCTR: Calibrating Trajectory Prediction for Uncertainty-Aware Motion Planning in Autonomous Driving (AAAI 2024 · similarity 0.84)
• Producing and Leveraging Online Map Uncertainty in Trajectory Prediction (CVPR 2024 · similarity 0.84)
• C2F-TP: A Coarse-to-Fine Denoising Framework for Uncertainty-Aware Trajectory Prediction (AAAI 2025 · similarity 0.84)
• BAT: Behavior-Aware Human-Like Trajectory Prediction for Autonomous Driving (AAAI 2024 · similarity 0.84)