Research map / Video / Action recognition
Viewpoint Rosetta Stone: Unlocking Unpaired Ego-Exo Videos for View-invariant Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns clip-level ego-exo view-invariant video representations by using a diffusion Rosetta Stone Translator trained on synchronized pairs to pseudo-align unpaired ego and exo videos, with soft contrastive alignment.
Problem
Learning view-invariant video representations across egocentric and exocentric viewpoints is limited by the scarcity of synchronized ego-exo paired data, while large unpaired ego and exo video collections remain underused.
Approach
A diffusion translator in feature space, trained on a moderate amount of synchronized ego-exo pairs, hallucinates exo features from ego features to pseudo-pair unpaired videos; a dual encoder then trains with contrastive losses that include RST-synthetic positives and textual-similarity soft weighting.
Key results
- ViewpointRosetta achieves 58.14 ego→exo and 47.21 exo→ego HR@5 in cross-view retrieval (avg 52.68), vs. 39.96 avg for SUM-L.
- Cross-view recognition top-1 accuracy of 34.47 vs. 24.83 for SUM-L (top-5 64.85 vs. 52.08).
- Ablation on HR@5 shows the RST and soft alignment both contribute; full model reaches 58.1 ego→exo versus 43.8 without RST and SL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VIEWPOINTROSETTA instead of the strongest baseline SUM-L | Cross-view retrieval on Ego-Exo4D · Ego-Exo4D cross-view retrieval, average | HR@5: 39.96 -> 52.68 | Helped |
| Use VIEWPOINTROSETTA instead of the strongest baseline LaViLa | Cross-view recognition on Ego-Exo4D · Ego-Exo4D cross-view recognition | top-1 accuracy: 26.43 -> 34.47 | Helped |
| Use VIEWPOINTROSETTA instead of the strongest baseline LaViLa | Cross-view recognition on Ego-Exo4D · Ego-Exo4D cross-view recognition | top-5 accuracy: 55.01 -> 64.85 | Helped |
| Use VIEWPOINTROSETTA instead of the strongest baseline SUM-L | Cross-view skill assessment on Ego-Exo4D · Ego-Exo4D cross-view skill assessment | accuracy: 55.10 -> 55.82 | Helped |
| Add RST feature translator (ablation) | VIEWPOINTROSETTA · Ego-Exo4D cross-view retrieval, ego→exo | HR@5: 44.4 -> 58.1 | Helped |
| Add RST feature translator (ablation) | VIEWPOINTROSETTA · Ego-Exo4D cross-view retrieval, exo→ego | HR@5: 30.7 -> 47.2 | Helped |
| Add soft alignment (ablation) | VIEWPOINTROSETTA · Ego-Exo4D cross-view retrieval, ego→exo | HR@5: 56.7 -> 58.1 | Helped |
| Add soft alignment (ablation) | VIEWPOINTROSETTA · Ego-Exo4D cross-view retrieval, exo→ego | HR@5: 46.9 -> 47.2 | Helped |
| Add RST feature translator and soft alignment (ablation) | VIEWPOINTROSETTA · Ego-Exo4D cross-view retrieval, ego→exo | HR@5: 43.8 -> 58.1 | Helped |
| Add RST feature translator and soft alignment (ablation) | VIEWPOINTROSETTA · Ego-Exo4D cross-view retrieval, exo→ego | HR@5: 29.1 -> 47.2 | Helped |
Limitations
- Skill assessment gains are small and most methods perform near chance.
- Relies on a moderate amount of synchronized ego-exo pairs to train the translator.
- Unpaired pseudo-pairs are noisy and rely on narration/text similarity for soft weighting.
Experimental setup
Datasets: Ego-Exo4D Ego4D HTM-AA HowTo100M
Benchmarks: Ego-Exo4D cross-view retrieval Ego-Exo4D cross-view keystep recognition Ego-Exo4D cross-view skill assessment
Metrics: HR@5 top-1 accuracy top-5 accuracy accuracy
Baselines: TimeSformer EgoVLP LaViLa Random Align ActorObserverNet VI Encoder EgoInstructor SUM-L
Builds on: DDPM DiT InfoNCE LaViLa SUM-L
Closest papers by meaning
- Bootstrap Your Own Views: Masked Ego-Exo Modeling for Fine-grained View-invariant Video Representations (CVPR 2025 · similarity 0.91)
- EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMs (NeurIPS 2025 · similarity 0.89)
- Retrieval-Augmented Egocentric Video Captioning (CVPR 2024 · similarity 0.89)
- Exocentric-to-Egocentric Video Generation (NeurIPS 2024 · similarity 0.87)
- Grounded-Exo2Ego: Structured Semantic Grounding for Robust Exocentric-to-Egocentric Video Generation (NeurIPS 2026 · similarity 0.87)
- Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.