Research map / 3D vision / 3D generation
Gaze Beyond the Frame: Forecasting Egocentric 3D Visual Span
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Forecasts a person's future 3D visual span (multi-level gaze/view volumes) by lifting SLAM keypoints into gaze-conditioned occupancy grids and predicting them with a 3D U-Net plus unidirectional transformer, on a newly curated FoVS-Aria/FoVS-EgoExo benchmark.
Problem
Predicting where a person's visual perception will be focused in their 3D surroundings in the future; existing gaze anticipation works in 2D image planes, which is ill-posed under self-motion.
Approach
EgoSpanLift converts SLAM semidense keypoints into multi-level volumetric visual-span occupancy grids (foveal, central, peripheral, head-orientation) using gaze cones, and an end-to-end 3D U-Net + unidirectional transformer forecasts the future union of these spans with a dice loss.
Key results
- On FoVS-Aria foveal span, the full model reaches IoU 0.2836 and F1 0.3709, versus 0.1987 IoU for the best baseline EgoChoir.
- On FoVS-EgoExo central span, the full model reaches IoU 0.4212 vs 0.1976 for EgoChoir.
- Projected back to 2D, the model reaches F1 0.515 on egocentric 2D gaze anticipation, matching CSTS (0.515) without 2D-specific training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| forecast 3D visual spans with EgoSpanLift and the full forecasting framework | FoVS-Aria · FoVS-Aria test split, Foveal 2° | IoU: 0.1987 -> 0.2836 | Helped |
| forecast 3D visual spans with EgoSpanLift and the full forecasting framework | FoVS-EgoExo · FoVS-EgoExo test split, Foveal 2° | IoU: 0.1556 -> 0.3692 | Helped |
| add previous-span information (ablation) | EgoSpanLift forecasting framework · FoVS-Aria test split, Foveal 2° | IoU: 0.0594 -> 0.2836 | Helped |
| add previous-span information (ablation) | EgoSpanLift forecasting framework · FoVS-EgoExo test split, Foveal 2° | IoU: 0.2255 -> 0.3692 | Helped |
| use Dice loss instead of binary cross-entropy (ablation) | EgoSpanLift forecasting framework · FoVS-Aria test split, Foveal 2° | IoU: 0.2059 -> 0.2836 | Helped |
| use Dice loss instead of binary cross-entropy (ablation) | EgoSpanLift forecasting framework · FoVS-EgoExo test split, Foveal 2° | IoU: 0.2870 -> 0.3692 | Helped |
| forecast multi-level spans jointly instead of using single-task models (ablation) | EgoSpanLift forecasting framework · FoVS-Aria test split, Foveal 2° | IoU: 0.2494 -> 0.2836 | Helped |
| add the global embedding (ablation) | EgoSpanLift forecasting framework · FoVS-Aria test split, Foveal 2° | IoU: 0.2624 -> 0.2836 | Helped |
| add the global embedding (ablation) | EgoSpanLift forecasting framework · FoVS-EgoExo test split, Foveal 2° | IoU: 0.3475 -> 0.3692 | Helped |
| use the full multi-level 3D forecasting framework instead of gaze-only prediction with EgoSpanLift (ablation) | EgoSpanLift forecasting framework · FoVS-Aria test split, Foveal 2° | IoU: 0.2494 -> 0.2836 | Helped |
| increase grid range to D=6.4 while maintaining 20cm precision (ablation) | EgoSpanLift forecasting framework · FoVS-EgoExo test split, Foveal 2° | IoU: 0.3692 -> 0.3640 | No clear effect |
| use 10cm precision with D=3.2 and R=32 (ablation) | EgoSpanLift forecasting framework · FoVS-EgoExo test split, Foveal 2° | IoU: 0.3692 -> 0.2493 | Hurt |
Surprising findings
- Baseline 2D gaze models lifted to 3D struggle on foveal span, while the 3D model trained without 2D-specific data matches 2D-specific models on 2D gaze anticipation.
Limitations
- Semidense SLAM keypoints cannot represent regions without keypoints, so foveal span is absent in about 20% of samples.
- Voxelization dominates latency, and the paper notes further optimization is needed for AR/VR hardware.
- Performance declines with longer future horizons, especially for foveal span.
Experimental setup
Datasets: Aria Everyday Activities Ego-Exo4D
Benchmarks: FoVS-Aria FoVS-EgoExo Aria-based egocentric 2D gaze anticipation
Metrics: 3D IoU F1 precision recall metric distance error
Baselines: OccFormer VoxFormer IAG EgoChoir GLC CSTS Global Prior
Builds on: 3D U-Net EgoChoir GLC CSTS Gaussian process head pose extrapolation
Closest papers by meaning
- EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Views (NeurIPS 2024 · similarity 0.89)
- EgoGen: An Egocentric Synthetic Data Generator (CVPR 2024 · similarity 0.89)
- GazeFlow: From Human Gaze Behavior to Generative Egocentric Gaze Prediction (NeurIPS 2026 · similarity 0.88)
- Gaze-VLM: Bridging Gaze and VLMs through Attention Regularization for Egocentric Understanding (NeurIPS 2025 · similarity 0.88)
- Vision-Guided Action: Enhancing 3D Human Motion Prediction with Gaze-informed Affordance in 3D Scenes (CVPR 2025 · similarity 0.87)
- EgoSim: An Egocentric Multi-view Simulator and Real Dataset for Body-worn Cameras during Motion and Activity (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.