Research map / Video / Video segmentation and tracking
Robust Ego-Exo Correspondence with Long-Term Memory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LM-EEC, a SAM 2-based ego-exo object correspondence model with a Memory-View Mixture-of-Experts routing module and a dual (ego/exo) memory bank with feature-level compression, achieving state-of-the-art on EgoExo4D.
Problem
Object-level correspondence between egocentric and exocentric videos is hard due to extreme viewpoint change, occlusion, and small objects; SAM 2 fuses ego and exo cues poorly and its FIFO memory loses long-term information on long videos.
Approach
Treat memory-aware and view-specific features as two experts fused by dense channel- and spatial-wise routing (MV-MoE) instead of simple addition, and maintain separate ego and exo memory banks compressed by merging the most temporally redundant feature points.
Key results
- On the EgoExo4D test set, LM-EEC reaches 54.98 IoU for Ego2Exo (vs. 52.13 for the base model and 38.26 for SimVOS) and 65.77 IoU for Exo2Ego (vs. 57.27 base model, 47.52 Cutie).
- Ablation on memory compression: the proposed feature-level compression achieves 0.5925 IoU vs. 0.5823 for FIFO on the validation set (Ego2Exo).
- For videos longer than 600 frames, association accuracy improves from 0.4724 (SAM 2 baseline) to 0.6150.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add LM-EEC's MV-MoE and memory strategy to the base model | SAM 2 base model · EgoExo4D test, Ego2Exo | IoU: 52.13 -> 54.98 | Helped |
| Add LM-EEC's MV-MoE and memory strategy to the base model | SAM 2 base model · EgoExo4D test, Ego2Exo | LE: 0.024 -> 0.017 | Helped |
| Add LM-EEC's MV-MoE and memory strategy to the base model | SAM 2 base model · EgoExo4D test, Ego2Exo | CA: 0.734 -> 0.778 | Helped |
| Add LM-EEC's MV-MoE and memory strategy to the base model | SAM 2 base model · EgoExo4D test, Ego2Exo | BA: 61.56 -> 64.22 | Hurt |
| Add LM-EEC's MV-MoE and memory strategy to the base model | SAM 2 base model · EgoExo4D test, Exo2Ego | IoU: 57.27 -> 65.77 | Helped |
| Add LM-EEC's MV-MoE and memory strategy to the base model | SAM 2 base model · EgoExo4D test, Exo2Ego | LE: 0.047 -> 0.031 | Helped |
| Add LM-EEC's MV-MoE and memory strategy to the base model | SAM 2 base model · EgoExo4D test, Exo2Ego | CA: 0.677 -> 0.774 | Helped |
| Add LM-EEC's MV-MoE and memory strategy to the base model | SAM 2 base model · EgoExo4D test, Exo2Ego | BA: 57.11 -> 58.14 | Helped |
| Replace FIFO, cluster, or IoU-selection compression with feature-point-level compression (ablation) | LM-EEC · EgoExo4D validation, Ego2Exo | IoU: 0.5823 -> 0.5925; 0.5867 -> 0.5925; 0.5880 -> 0.5925 | Helped |
| Replace FIFO, cluster, or IoU-selection compression with feature-point-level compression (ablation) | LM-EEC · EgoExo4D validation, Ego2Exo | LE: 0.0214 -> 0.0198; 0.0208 -> 0.0198; 0.0204 -> 0.0198 | Helped |
| Replace FIFO, cluster, or IoU-selection compression with feature-point-level compression (ablation) | LM-EEC · EgoExo4D validation, Ego2Exo | CA: 0.7874 -> 0.8006; 0.7942 -> 0.8006; 0.7948 -> 0.8006 | Helped |
| Add MV-MoE (ablation) | STCN · EgoExo4D test, Ego2Exo | IoU: 27.39 -> 30.51 | Helped |
Surprising findings
- XSegTx achieves the highest balanced accuracy (BA) on both query directions, and LM-EEC does not top BA, which the authors attribute to failing to handle object disappearance.
Limitations
- Does not explicitly handle object disappearance, leading to segmentation of visually similar background objects and lower balanced accuracy.
- Inherits SAM 2's lack of explicit object presence prediction.
Experimental setup
Datasets: EgoExo4D
Benchmarks: EgoExo4D ego-exo correspondence test set
Metrics: IoU LE CA BA association accuracy
Baselines: XSegTx XView-XMem (w/ finetuning) XView-XMem (+ XSegTx) STCN QDMN GSFM SimVOS Cutie SAM 2 base model
Builds on: SAM 2 XView-XMem XMem STCN QDMN
Closest papers by meaning
- SAM-I2V: Upgrading SAM to Support Promptable Video Segmentation with Less than 0.2% Training Cost (CVPR 2025 · similarity 0.87)
- SAM 2: Segment Anything in Images and Videos (ICLR 2025 · similarity 0.87)
- Robust Egocentric Referring Video Object Segmentation via Dual-Modal Causal Intervention (NeurIPS 2025 · similarity 0.85)
- Bootstrap Your Own Views: Masked Ego-Exo Modeling for Fine-grained View-invariant Video Representations (CVPR 2025 · similarity 0.85)
- Exocentric-to-Egocentric Video Generation (NeurIPS 2024 · similarity 0.84)
- Grounded-Exo2Ego: Structured Semantic Grounding for Robust Exocentric-to-Egocentric Video Generation (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.