Research map / Vision-language and multimodal / Human and embodied modeling
Ego4o: Egocentric Human Motion Capture and Understanding from Multi-Modal Input
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Ego4o is a multi-modal framework that fuses 1-3 sparse IMUs, an optional egocentric image, and optional motion text into a part-aware VQ-VAE latent space for motion capture, and uses a LLaVA-based motion LLM to generate motion descriptions that feed back to improve capture.
Problem
Human motion capture and understanding from consumer wearables (VR/AR headsets, smart glasses, phones, smartwatches) where input modalities such as egocentric images, sparse IMUs, and text are intermittently available and individually limited.
Approach
A part-aware motion VQ-VAE whose codebook is shared by a transformer multi-modal encoder that takes any subset of IMUs, image, and text (trained with random modality masking); the resulting motion codes are also fed to a fine-tuned multi-modal LLM to generate descriptions, which are fed back to improve capture.
Key results
- On Nymeria, Ego4o with IMUs, image and ground-truth text reaches 84.82 mm MPJPE versus 105.7 mm for IMUPoser; IMU-only Ego4o-IMU reaches 95.86 mm.
- Motion description generation on Nymeria reaches BERTScore 30.13 vs 14.09 for MotionGPT and RougeL 38.95 vs 32.33, though BLEU@4 is lower (7.46 vs 10.31).
- Using generated descriptions as input improves capture to 88.65 mm MPJPE (w/ gen text) versus 95.86 mm for IMU-only.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add egocentric images and ground-truth motion descriptions to 1–3 IMUs | Ego4o-IMU · Nymeria | MPJPE (mm): 95.86 -> 84.82 | Helped |
| Use Ego4o instead of IMUPoser with 1–3 IMUs | IMU-based motion capture · Nymeria | MPJPE (mm): 105.7 -> 95.86 | Helped |
| Use Ego4o for motion description generation instead of MotionGPT | Motion description generation · Nymeria | Bert(idf): 14.09 -> 30.13 | Helped |
| Use Ego4o instead of MotionGPT for description generation | Motion description generation · Nymeria | Bleu@4: 10.31 -> 7.46 | Hurt |
| Disable test-time optimization (ablation) | Ego4o-IMU · Nymeria | MPJPE (mm): 84.82 -> 85.93 | Hurt |
| Add ground-truth motion text to IMUs, without image input (ablation) | Ego4o-IMU · Nymeria | MPJPE (mm): 95.86 -> 86.22 | Helped |
| Add an egocentric image to IMUs, without text input (ablation) | Ego4o-IMU · Nymeria | MPJPE (mm): 95.86 -> 90.81 | Helped |
| Add generated motion text to IMUs (ablation) | Ego4o-IMU · Nymeria | MPJPE (mm): 95.86 -> 88.65 | Helped |
| Add an egocentric image and generated motion text to IMUs (ablation) | Ego4o-IMU · Nymeria | MPJPE (mm): 95.86 -> 87.00 | Helped |
| Remove egocentric image input (ablation) | Ego4o motion understanding · Nymeria | Bert(idf): 30.13 -> 22.44 | Hurt |
| Remove motion-token input (ablation) | Ego4o motion understanding · Nymeria | Bert(idf): 30.13 -> 25.55 | Hurt |
| Use ground-truth motion codes instead of predicted motion tokens (ablation) | Ego4o motion understanding · Nymeria | Bert(idf): 30.13 -> 31.38 | Helped |
Surprising findings
- Ego4o's generated descriptions are not better on BLEU@4 than MotionGPT, which the authors attribute to BLEU not capturing semantic quality.
- Test-time optimization lowers MPJPE but increases jitter relative to no optimization, attributed to noisy IMU signals.
Limitations
- Requires motion sequences as input, introducing latency for online applications.
- Multi-round conversational capability is limited.
- Unable to compare against some related works (MobilePoser, Diffusion-Poser) due to lack of code.
Experimental setup
Datasets: Nymeria AMASS DIP-IMU HumanML3D
Benchmarks: DIP-IMU Nymeria
Metrics: MPJPE PA-MPJPE jitter BERTScore BLEU ROUGE-L
Baselines: IMUPoser DIP TransPose TM2T MotionGPT
Builds on: VQ-VAE TLControl part-aware VQ-VAE LLaVA IMUPoser CLIP
Closest papers by meaning
- EgoLM: Multi-Modal Language Model of Egocentric Motions (CVPR 2025 · similarity 0.89)
- EMHI: A Multimodal Egocentric Human Motion Dataset with HMD and Body-Worn IMUs (AAAI 2025 · similarity 0.89)
- EgoSim: An Egocentric Multi-view Simulator and Real Dataset for Body-worn Cameras during Motion and Activity (NeurIPS 2024 · similarity 0.88)
- FRAME: Floor-aligned Representation for Avatar Motion from Egocentric Video (CVPR 2025 · similarity 0.88)
- Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives (CVPR 2024 · similarity 0.87)
- EgoGen: An Egocentric Synthetic Data Generator (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.