Research map / Video / Video-language understanding
EgoLM: Multi-Modal Language Model of Egocentric Motions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EgoLM, a decoder-only LLM (GPT-2) that jointly models egocentric motions and language conditioned on sparse motion sensors and egocentric video, unifying motion tracking and narration via PQ-VAE tokenization, motion-augmented pre-training, and multi-task instruction tuning.
Problem
Egocentric motion understanding from wearable devices is under-constrained because the wearer's body is rarely visible in video and sparse motion sensors capture only a few body parts; aligning continuous motion with language is also hard.
Approach
Treat motion as a language by tokenizing it with a product-quantization VAE, augment a pre-trained decoder-only LM with motion tokens and motion-regression-aware next-token pre-training, then jointly instruction-tune on motion tracking from sensors+video, motion narration, text-to-motion, and motion-to-text.
Key results
- Three-points motion tracking with egocentric video reduces full-body joint error from 83.88 mm to 73.38 mm.
- One-point tracking with video reduces joint error by about 20 mm (127.45 to 106.95 mm full-body).
- Joint training on motion narration from three-points and video reaches BERTScore 19.40 and BLEU@4 12.74, vs. MotionGPT at 14.09 and 10.31 for motion-only narration.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add egocentric video to 3pts tracking | EgoLM · Nymeria test, 3pts motion tracking | Full joint position error (mm): 83.88 -> 73.38 | Helped |
| Add egocentric video to 1pt tracking | EgoLM · Nymeria test, 1pt motion tracking | Full joint position error (mm): 127.45 -> 106.95 | Helped |
| Jointly train with motion, video, and motion-to-text tasks | EgoLM · Nymeria test, motion narration from motion + video | Bert: 19.97 -> 20.32 | Helped |
| Increase tracking window from 60 to 120 frames (ablation) | EgoLM without video · Nymeria test, 3pts tracking | Full joint position error (mm): 83.88 -> 79.61 | Helped |
| Increase tracking window from 60 to 120 frames (ablation) | EgoLM with video · Nymeria test, 3pts tracking + video | Full joint position error (mm): 73.38 -> 72.76 | Helped |
| Add video rather than only increasing the window to 120 frames (ablation) | EgoLM 3pts tracking · Nymeria test, 3pts tracking | Full joint position error (mm): 79.61 -> 73.38 | Helped |
| Use product quantization in the motion tokenizer (ablation) | Motion PQ-VAE · Motion reconstruction | MPJPE (mm): 51.60 -> 39.63 | Helped |
| Increase codebook entries and reduce code dimension (ablation) | Motion PQ-VAE · Motion reconstruction | MPJPE (mm): 39.63 -> 39.13 -> 34.49 | Helped |
| Scale backbone from GPT-2 Medium to GPT-2 Large (ablation) | EgoLM TPV2T · Nymeria test, motion narration from 3pts + video | Bert: 18.38 -> 19.56 | Helped |
| Add motion regression loss Lreg to next-token prediction (ablation) | EgoLM · Nymeria test, 3pts tracking + video | Full joint position error (mm): 74.10 -> 73.38 | Helped |
Surprising findings
- Adapting AvatarPoser with concatenated CLIP embeddings failed to exploit video input, worsening error (127.08 mm vs 85.89 mm).
Limitations
- Motion tokenizer reconstruction errors bound tracking performance.
- CLIP compresses each video frame into a single vector, making object interaction identification hard.
- Hallucination problem in generated narrations.
Experimental setup
Datasets: Nymeria
Benchmarks: Nymeria
Metrics: MPJPE full/upper/lower body (mm) joint angle error BERTScore BLEU@1 BLEU@4 ROUGE-L
Baselines: AvatarPoser BoDiffusion EgoEgo TM2T MotionGPT
Builds on: MotionGPT LLaVA VQ-VAE AvatarPoser product quantization
Closest papers by meaning
- Ego4o: Egocentric Human Motion Capture and Understanding from Multi-Modal Input (CVPR 2025 · similarity 0.89)
- Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos (CVPR 2025 · similarity 0.87)
- EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT (NeurIPS 2025 · similarity 0.87)
- EgoSim: An Egocentric Multi-view Simulator and Real Dataset for Body-worn Cameras during Motion and Activity (NeurIPS 2024 · similarity 0.86)
- Action Scene Graphs for Long-Form Understanding of Egocentric Videos (CVPR 2024 · similarity 0.86)
- EgoProx: Evaluating MLLMs on Egocentric 3D Proximity Reasoning Across a Cognitive Hierarchy (CVPR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.