Research map / Video / Video-language understanding
EgoDTM: Towards 3D-Aware Egocentric Video-Language Pretraining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EgoDTM, an egocentric video-language model that adds a lightweight depth-prediction decoder (supervised by pseudo-depth from Depth Anything V2) and LLM-enriched hand-object captions built with a detect-track-generate pipeline.
Problem
Egocentric video-language pretraining relies on 1D text or 2D region cues, lacking 3D spatial awareness; the paper asks whether depth-based 3D-aware pretraining can improve egocentric video representations.
Approach
A lightweight 3D-aware decoder (plain FPN, depth-aware transformer decoder with adaptive bins) predicts low-resolution depth from video features during video-text contrastive pretraining, combined with spatially enriched captions generated by an LLM guided by temporally consistent HOI masks (HOID detection + SAM2 tracking).
Key results
- EgoDTM reaches 33.5% mAP average on EK100-MIR zero-shot retrieval, versus 32.9% for AVION.
- On the H2O depth estimation task, EgoDTM reports scale-aware RMSE 0.539, about 9.8% better than the second-best LaViLa.
- On Franka Kitchen robot manipulation, EgoDTM averages 58.6% success, versus 52% for LaViLa and 38.4% for CLIP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add depth estimation loss to video-text contrastive pretraining (ablation) | EgoDTM with L_vtc · EK100MIR | mAP: 29.7 -> 31.3 | Helped |
| add enriched-text contrastive learning to L_vtc + L_depth (ablation) | EgoDTM · EK100MIR | mAP: 31.3 -> 33.1 | Helped |
| add depth estimation loss to L_augvtc (ablation) | EgoDTM · EK100MIR | mAP: 31.3 -> 33.1 | Helped |
| use 8 rather than 4 depth queries (ablation) | EgoDTM · EK100MIR | mAP: 32.6 -> 33.1 | Helped |
| increase depth queries from 8 to 16 (ablation) | EgoDTM · EK100MIR | mAP: 33.1 -> 32.3 | Hurt |
| increase depth-map resolution from 28p to 112p (ablation) | EgoDTM · EK100MIR | mAP: 30.4 -> 31.2 | Helped |
| increase model size from Base to Large (ablation) | EgoDTM · EK100MIR | mAP: 33.1 -> 36.4 | Helped |
| randomly substitute original and enriched captions instead of replacing all original captions (ablation) | EgoDTM · EK100MIR | mAP: 32.8 -> 33.1 | Helped |
| increase batch size from 1024 to 4096 (ablation) | EgoDTM · EK100MIR | mAP: 32.4 -> 33.5 | Helped |
| pretrain EgoDTM rather than AVION | EgoDTM · EK100MIR | mAP: 32.9 -> 33.5 | Helped |
| pretrain EgoDTM rather than AVION | EgoDTM · EGTEA | mean: 38.6 -> 40.2 | Helped |
| pretrain EgoDTM rather than AVION | EgoDTM · EgoMCQ | Intra: +1.5% | Helped |
Surprising findings
- Depth pretraining slightly hurt EgoNLQ performance without the enriched-text objective, according to the authors.
- R3M and MPI outperform EgoDTM on several robot manipulation tasks such as open microwave, which the authors attribute to overfitting to real-world scenarios.
Limitations
- Generalization to broader indoor scenarios is limited, as the model is tuned to hand-object interaction scenes.
- The HOI mask pipeline reaches 43.02 mAP versus 60.70 for the supervised upper bound on VISOR, so pseudo-labels are noisy.
Experimental setup
Datasets: Ego4D EK100 EGTEA H2O VISOR
Benchmarks: EK100-MIR EgoMCQ EGTEA EK100-CLS H2O depth estimation Franka Kitchen EgoNLQ EgoMQ VISOR
Metrics: mAP nDCG accuracy top-1 accuracy threshold accuracy RMSE success rate mIoU R@n IoU
Baselines: EgoVLP EgoVLPv2 LaViLa AVION HelpingHands HENASY CLIP R3M
Builds on: EgoVLP CLIP Depth Anything V2 SAM2 AVION
Closest papers by meaning
- Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding (CVPR 2025 · similarity 0.88)
- Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors (NeurIPS 2025 · similarity 0.87)
- Vid-LLM: A Compact Video-based 3D Multimodal LLM with Reconstruction–Reasoning Synergy (ICLR 2026 · similarity 0.87)
- MLLMs Need 3D-Aware Representation Supervision for Scene Understanding (NeurIPS 2025 · similarity 0.87)
- Empowering Large Language Models with 3D Situation Awareness (CVPR 2025 · similarity 0.87)
- Retrieval-Augmented Egocentric Video Captioning (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.