Research map / 3D vision / Depth and stereo
Fusing Personal and Environmental Cues for Identification and Segmentation of First-Person Camera Wearers in Third-Person Views
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PersonEnvironmentNet (PEN), fusing personal (individual matching) and environmental (GlueStick line-matching geometry) branches to identify and segment first-person camera wearers in third-person views, plus the TF2023 dataset and an EgoRate overfitting metric.
Problem
Identifying and segmenting camera wearers in a third-person (exocentric) view given synchronized first-person (egocentric) views, without past-frame information, in scenes with multiple wearers; prior models can overfit to the third-person input.
Approach
A three-module model: a personal branch relating individuals visible in the ego view to the exo candidate via multi-head attention, an environmental branch matching background lines (GlueStick) with candidate masks, and a self-attention fusion head; plus "ghost samples" labeled 0.5 to discourage appearance memorization, and EgoRate, which measures reliance on the ego view by replacing ego input wi
Key results
- On TF2023, PEN (self-attention fusion) reaches 0.44 IoU, 52.9% accuracy and 0.97 IoU-EgoRate, versus 0.33 IoU / 39.9% for the ViT two-stream baseline.
- On IUShareView, the two-stream ViT baseline reaches 99.3% accuracy with EgoRate 0.0, while PEN reaches 99.7% accuracy with EgoRate 0.77.
- Adding ghost samples to the environmental branch raises IoU-EgoRate from 0.67 to 0.95 without substantially changing IoU or accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine personal and environmental branches with self-attention | PersonEnvironmentNet · TF2023 | IoU: 0.40 -> 0.44 | Helped |
| Combine personal and environmental branches with self-attention | PersonEnvironmentNet · TF2023 | Acc: 48.9 -> 52.9 | Helped |
| Replace the two-stream ViT baseline with PEN | Two-stream ViT baseline · TF2023 | IoU: 0.33 -> 0.44 | Helped |
| Replace the two-stream ViT baseline with PEN | Two-stream ViT baseline · TF2023 | Acc: 39.9 -> 52.9 | Helped |
| Replace the two-stream ViT baseline with PEN | Two-stream ViT baseline · IUShareView | EgoRate (Acc): 0.0 -> 0.77 | Helped |
| Use the personal branch alone instead of fusing both branches (ablation) | PersonEnvironmentNet · TF2023 | IoU: 0.44 -> 0.40 | Hurt |
| Use the environmental branch alone instead of fusing both branches (ablation) | PersonEnvironmentNet · TF2023 | IoU: 0.44 -> 0.39 | Hurt |
| Fuse branch outputs by addition rather than self-attention (ablation) | PersonEnvironmentNet · TF2023 | Acc: 52.5 vs 52.9 | No clear effect |
| Fuse branch outputs by concatenation rather than self-attention (ablation) | PersonEnvironmentNet · TF2023 | Acc: 52.4 vs 52.9 | No clear effect |
| Use BCE loss instead of contrastive loss (ablation) | Two-Stream baseline · TF2023 | Acc: 39.9 -> 31.9 | Hurt |
| Replace ViT with Swin in the two-stream baseline (ablation) | Two-Stream baseline · TF2023 | Acc: 39.9 -> 36.9 | Hurt |
| Add ghost samples during environmental-branch training (ablation) | Environmental branch · TF2023 | EgoRate (Acc): 0.65 -> 0.95 | Helped |
Surprising findings
- A two-stream ViT baseline reaches 99.3% accuracy on IUShareView while ignoring the first-person input entirely (EgoRate 0), indicating overfitting to the third-person view.
- Swin-based two-stream baselines score lower EgoRate than ViT, and the BCE variant of the baseline performs worse than contrastive on Acc.
Limitations
- Candidate masks come from a pretrained Mask R-CNN with score threshold 0.9, so segmentation misses or over-segmentation lower absolute scores.
- Fusion variants differ only slightly, and the matching model may fail under large viewpoint differences such as 180-degree opposite views.
- Absolute accuracy on TF2023 remains modest (about 53%).
Experimental setup
Datasets: TF2023 IUShareView
Benchmarks: TF2023 IUShareView
Metrics: IoU Accuracy EgoRate
Baselines: Third-first Two-stream baseline (ViT) Two-stream baseline (Swin) Random guess
Builds on: GlueStick Mask R-CNN third-first model Vision Transformer
Closest papers by meaning
- EgoSim: An Egocentric Multi-view Simulator and Real Dataset for Body-worn Cameras during Motion and Activity (NeurIPS 2024 · similarity 0.86)
- The 3D-PC: a benchmark for visual perspective taking in humans and machines (ICLR 2025 · similarity 0.84)
- Instance Tracking in 3D Scenes from Egocentric Videos (CVPR 2024 · similarity 0.83)
- EgoPrivacy: What Your First-Person Camera Says About You? (ICML 2025 · similarity 0.83)
- Learning from Synchronization: Self-Supervised Uncalibrated Multi-View Person Association in Challenging Scenes (CVPR 2025 · similarity 0.83)
- 3D Human Pose Perception from Egocentric Stereo Videos (CVPR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.