Research map / Video / Video-language understanding
Bootstrap Your Own Views: Masked Ego-Exo Modeling for Fine-grained View-invariant Video Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BYOV, a masked ego-exo modeling framework (self-view and cross-view frame-token masked prediction over a frozen CLIP ViT-B/16) that learns fine-grained view-invariant video representations from unpaired ego-exo videos.
Problem
Learning fine-grained view-invariant video representations from unpaired and asynchronous egocentric and exocentric videos, where ego and exo views differ in perspective, motion, and context.
Approach
Combines masked self-view modeling (causal prediction of masked frame-token embeddings within each view) with masked cross-view modeling (predicting one view's tokens from the other view's latents with a high masking ratio), plus a selective token merging step that keeps tokens with large frame-to-frame change instead of using hand-object detectors.
Key results
- On Break Eggs, BYOV reaches 74.30 regular classification F1 vs 66.23 for AE2 and 0.8533 phase progression vs 0.5109 for AE2.
- On Charades-Ego, BYOV achieves 31.8 mAP in regular linear probing vs 13.7 for CLIP features (ViT-B/16).
- Ablations on Break Eggs show removing MSM drops F1 from 73.53 to 62.12 and removing the causal mask drops phase progression from 0.8533 to 0.6420.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BYOV instead of AE2 | Fine-grained ego-exo video representations · Break Eggs | Phase progression: 0.5109 -> 0.8533 | Helped |
| Use BYOV instead of AE2 | Fine-grained ego-exo video representations · Pour Milk | Kendall’s τ: 0.9062 -> 0.9466 | Helped |
| Use BYOV instead of AE2 | Fine-grained ego-exo video representations · Pour Liquid | Phase progression: 0.1380 -> 0.4483 | Helped |
| Use BYOV instead of AE2 | Fine-grained ego-exo video representations · Tennis Forehand | F1 score (Ego2Exo): 84.71 -> 94.47 | Helped |
| Add token selection (ablation) | BYOV · Break Eggs | F1 score: 71.84 -> 73.53 | Helped |
| Add a causal mask to the MSM decoder (ablation) | BYOV · Break Eggs | Phase progression: 0.6420 -> 0.8533 | Helped |
| Add masked self-view modeling (MSM) (ablation) | BYOV · Break Eggs | Phase progression: 0.4362 -> 0.8533 | Helped |
| Add masked cross-view modeling (MCM) (ablation) | BYOV · Break Eggs | F1 score: 57.43 -> 73.53 | Helped |
Surprising findings
- Frozen ImageNet and CLIP features perform worse than random features on phase progression and Kendall's tau in cross-view settings, showing pretrained frame embeddings alone do not capture action progress.
Limitations
- Relies on a frozen image encoder, so temporal dynamics are learned only on top of frame-level features.
- Evaluated only on the AE2 benchmark's four tasks and Charades-Ego, limiting breadth of evaluation.
Experimental setup
Datasets: AE2 ego-exo benchmark (Break Eggs, Pour Milk, Pour Liquid, Tennis Forehand) Charades-Ego
Benchmarks: AE2 benchmark (Break Eggs, Pour Milk, Pour Liquid, Tennis Forehand) Charades-Ego
Metrics: F1 score mAP@10 R-square (phase progression) Kendall's tau mAP
Baselines: Random features ImageNet features CLIP features ActorObserverNet TCN CARL TCC GTA AE2
Builds on: VideoMAE MaskFeat BERT AE2 CLIP
Closest papers by meaning
- Viewpoint Rosetta Stone: Unlocking Unpaired Ego-Exo Videos for View-invariant Representation Learning (CVPR 2025 · similarity 0.91)
- EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMs (NeurIPS 2025 · similarity 0.89)
- EgoExo-Gen: Ego-centric Video Prediction by Watching Exo-centric Videos (ICLR 2025 · similarity 0.87)
- Exocentric-to-Egocentric Video Generation (NeurIPS 2024 · similarity 0.87)
- Retrieval-Augmented Egocentric Video Captioning (CVPR 2024 · similarity 0.87)
- Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.