Research map / 3D vision / 3D generation
Learning to Generate Diverse Pedestrian Movements from Web Videos with Noisy Labels
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds CityWalkers, a web-video dataset of pedestrian movements with pseudo-labels, and PedGen, a diffusion model with iterative anomaly-based label filtering, mask embeddings for partial labels, and a 3D voxel context encoder for generating context-aware pedestrian movements.
Problem
Generating diverse, realistic pedestrian movements conditioned on scene context, body shape, and goal position, when training labels come from noisy pseudo-labels on web videos.
Approach
Learn pedestrian movement generation from noisy web-video pseudo-labels by iteratively filtering anomalous labels via reconstruction error of a diffusion model, training on partial labels with a learnable mask embedding, and encoding scene context by lifting 2D depth and semantics into a local 3D voxel grid.
Key results
- Without goal condition, PedGen reaches 4.08 aADE on CityWalkers vs 4.55 (MDM) and 4.67 (HumanMAC), and 1.6% collision rate on CARLA vs 2.1% (MDM) and 2.5% (HumanMAC).
- With goal condition, PedGen reaches 1.08 aADE on CityWalkers vs 1.26 for TRUMANS, and 0.0% CR and FFR on CARLA vs 0.6% for TRUMANS.
- Goal point is the most important context factor, reducing aADE by 72.9% (4.08 to 1.09 range in ablation); scene plus human plus goal yields 0.96 aADE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PedGen instead of MDM without goal conditioning | Pedestrian movement generation · CityWalkers validation, no goal | aADE: 4.55 -> 4.08 | Helped |
| Use PedGen instead of HumanMAC without goal conditioning | Pedestrian movement generation · Waymo test, no goal | aADE: 5.29 -> 5.15 | Helped |
| Use PedGen instead of TRUMANS with goal conditioning | Pedestrian movement generation · CityWalkers validation, goal | aADE: 1.26 -> 1.08 | Helped |
| Add automatic filtering of anomaly labels (ablation) | PedGen without context · CityWalkers validation | aADE: 4.45 -> 4.32 | Helped |
| Add partial labels to training (ablation) | PedGen without context · CityWalkers validation | aADE: 4.45 -> 4.08 | Helped |
| Add scene context (ablation) | PedGen without other context · CityWalkers validation | aADE: 4.08 -> 3.75 | Helped |
| Add human characteristics context (ablation) | PedGen without other context · CityWalkers validation | aADE: 4.08 -> 3.24 | Helped |
| Add goal conditioning (ablation) | PedGen without context · CityWalkers validation | aADE: 4.08 -> 1.09 | Helped |
| Add scene and human context alongside goal conditioning (ablation) | PedGen with goal · CityWalkers validation | aADE: 1.09 -> 0.96 | Helped |
| Train on CityWalkers rather than SLOPER4D (ablation) | PedGen without context · CityWalkers validation | aADE: 6.04 -> 4.08 | Helped |
| Use full rather than half of the CityWalkers training data (ablation) | PedGen without context · CityWalkers validation | aADE: 4.19 -> 4.08 | Helped |
| Add trajectory and geometry losses (ablation) | PedGen without context · CityWalkers validation | aADE: 4.47 -> 4.08 | Helped |
Surprising findings
- Training on CityWalkers with pseudo-labels greatly outperforms training on the cleaner SLOPER4D dataset (4.08 vs 6.04 aADE without context).
- Human-context training on SLOPER4D overfits the 12 subjects, degrading mADE to 3.82 versus 1.09 with CityWalkers.
Limitations
- Only static scene context at the starting frame is modeled; dynamic context such as other pedestrians' history is ignored.
- Only single-pedestrian movements are generated; group activities are not modeled.
- Pseudo-label noise from WHAM, depth, and segmentation models remains inevitable in CityWalkers.
Experimental setup
Datasets: CityWalkers SLOPER4D Waymo Open Dataset CARLA
Benchmarks: CityWalkers validation set Waymo test set CARLA test set
Metrics: mADE aADE mFDE aFDE Collision Rate Foot Floating Rate
Baselines: MDM HumanMAC TRUMANS
Builds on: classifier-free guidance DDIM MDM WHAM
Closest papers by meaning
- A Unified Environmental Network for Pedestrian Trajectory Prediction (AAAI 2024 · similarity 0.87)
- Towards Predicting Any Human Trajectory In Context (NeurIPS 2025 · similarity 0.86)
- MotionMix: Weakly-Supervised Diffusion for Controllable Motion Generation (AAAI 2024 · similarity 0.85)
- Move-in-2D: 2D-Conditioned Human Motion Generation (CVPR 2025 · similarity 0.85)
- Motion Diversification Networks (CVPR 2024 · similarity 0.85)
- TrajCLIP: Pedestrian trajectory prediction method using contrastive learning and idempotent networks (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.