Research map / 3D vision / 3D generation
Text to Blind Motion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces BlindWays, an IMU-based 3D motion and text-annotation benchmark of 11 blind/low-vision pedestrians navigating urban routes, and shows that state-of-the-art text-to-motion and motion-prediction models generalize poorly to blind motion.
Problem
Existing 3D human motion datasets and models lack diversity and under-represent people with disabilities, such as blind pedestrians, whose motion may be less predictable to autonomous-vehicle motion models.
Approach
A multimodal benchmark pairing Xsens IMU-based motion capture (including a mobility-aid joint) of blind pedestrians in real urban routes with high- and low-level text descriptions and egocentric and third-person video, used to test how well existing text-to-motion and motion-prediction models generalize to blind motion.
Key results
- Models trained on BlindWays reach FID 3.340 (HumanML3D) and 5.101 (MotionGPT) versus 11.203 and 15.002 when trained on Motion-X only.
- In motion-conditioned prediction, MDN achieves APD 15.14 versus 7.68 for CVAE, with ADE 0.45 and FDE 0.56.
- Pre-training on Motion-X then fine-tuning on BlindWays underperforms BlindWays-only training on arm and aid joints (e.g., arm FDE 0.99 vs 0.77 for HumanML3D).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train HumanML3D on BlindWays instead of Motion-X | HumanML3D · BlindWays text-to-motion | FID: 11.203 -> 3.340 | Helped |
| Train MotionGPT on BlindWays instead of Motion-X | MotionGPT · BlindWays text-to-motion | R Top1: 0.046 -> 0.054 | Helped |
| Pre-train on Motion-X, then train on BlindWays | HumanML3D · BlindWays text-to-motion | MModality: 1.896 ± 0.037 -> 4.921 ± 0.051 | Helped |
| Pre-train on Motion-X, then train on BlindWays | HumanML3D · BlindWays text-to-motion | FID: 3.340 ± 0.257 -> 8.612 ± 0.480 | Hurt |
| Use DLow instead of CVAE | Motion-conditioned prediction · BlindWays | FDE: 0.56 to 0.59 | Hurt |
| Use MDN instead of CVAE | Motion-conditioned prediction · BlindWays | ADE: 0.47 to 0.45 | Helped |
Surprising findings
- Pre-training on Motion-X followed by fine-tuning on BlindWays gives mixed results and can hurt arm and mobility-aid joint accuracy compared with training from scratch on BlindWays.
- Off-the-shelf text-to-motion models trained on Motion-X generate diverse but unrealistic blind motions.
Limitations
- Small sample of 11 participants (mostly blind, one low-vision) limits diversity of physical characteristics.
- Motion-capture suit is expensive (about $6,500), limiting larger-scale studies.
- Filtering of noisy IMU sequences and manual inspection were needed for tracking failures.
Experimental setup
Datasets: BlindWays Motion-X HumanML3D
Benchmarks: BlindWays
Metrics: R Top1 FID DIV MModality ADE FDE APD NPSS NDMS
Baselines: HumanML3D MotionGPT CVAE DLow MDN Zero Velocity
Builds on: HumanML3D MotionGPT MDM DLow
Closest papers by meaning
- mmWalk: Towards Multi-modal Multi-view Walking Assistance (NeurIPS 2025 · similarity 0.84)
- Harmonizing Stochasticity and Determinism: Scene-responsive Diverse Human Motion Prediction (NeurIPS 2024 · similarity 0.82)
- EgoBlind: Towards Egocentric Visual Assistance for the Blind (NeurIPS 2025 · similarity 0.82)
- Instantaneous Perception of Moving Objects in 3D (CVPR 2024 · similarity 0.82)
- Pedestrian Trajectory Prediction with Missing Data: Datasets, Imputation, and Benchmarking (NeurIPS 2024 · similarity 0.82)
- Vision-Guided Action: Enhancing 3D Human Motion Prediction with Gaze-informed Affordance in 3D Scenes (CVPR 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.