Research map / Detection and segmentation / CNN features
Foveated Instance Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FovealSeg, a gaze-driven foveated instance segmentation framework (FSNet) that segments only the instance under the user's gaze on AR/VR devices, reusing masks across frames with similar gaze, cutting FLOPs and latency.
Problem
Instance segmentation is computationally expensive on resource-constrained AR/VR devices at high input resolution, causing latency that degrades real-time user experience.
Approach
Uses real-time gaze location to segment only the instance of interest (IOI) via a gaze-aware saliency-guided downsampler (FSNet) feeding a segmentation network with binary-mask and classification branches, and uses saccade/fixation detection plus frame-similarity checks to skip or reuse segmentation across frames.
Key results
- FSNet+SegFormer-B5 reaches IoU 0.58 on Aria and 0.54-0.55 on LVIS/ADE20K range, outperforming Avg-subsampling baselines by at least 0.14 IoU.
- FovealSeg achieves up to 1.96x FLOPs reduction versus no-skip and up to 75x versus no-downsampling baseline.
- On GPGPU-Sim-simulated Jetson Orin NX, FSNet+Seg-B5 runs at 84 ms versus 1860 ms for SegFormer-B5 (over 20x faster).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FSNet instead of uniform subsampling or LTD | DeepLab-based instance segmentation · Cityscapes | IoU: 0.37 -> 0.52 | Helped |
| Use FSNet instead of uniform subsampling or LTD | HRNet-based instance segmentation · ADE20K | IoU: 0.41 -> 0.56 | Helped |
| Use FSNet instead of uniform subsampling or LTD | SegFormer-B4-based instance segmentation · LVIS | IoU′: 0.41 -> 0.56 | Helped |
| Use FSNet instead of uniform subsampling or LTD | SegFormer-B5-based instance segmentation · Aria Everyday Activities | IoU: 0.41 -> 0.58 | Helped |
| Use FovealSeg rather than full-resolution SegFormer-B5 processing | Cityscapes sequential-frame segmentation · Cityscapes sequential frames | latency: 1860 ms -> 84 ms | Helped |
| Reduce input resolution to 32 × 64 (ablation) | FSNet+DeepLab · Cityscapes | IoU: 0.52 -> 0.36 | Hurt |
| Increase Gaussian sampler kernel size from 17 to 33 (ablation) | FSNet+DeepLab · Cityscapes | IoU: 0.48 -> 0.52 | Helped |
| Replace gaze information with random noise (ablation) | FSNet+DeepLab · Cityscapes | IoU: 0.52 -> 0.14 | Hurt |
Surprising findings
- Replacing gaze input with random noise drops FSNet IoU by over 0.3 on Cityscapes, showing gaze is critical.
Limitations
- Efficiency results come from GPGPU-Sim simulation rather than measured on real AR/VR hardware.
- Gaze locations for Cityscapes are synthetically injected from Aria Everyday gaze traces, since Cityscapes lacks gaze data.
- Performance drops substantially at low downsampled resolution (e.g., IoU 0.52 to 0.36 at 32x64).
Experimental setup
Datasets: Cityscapes ADE20K LVIS Aria Everyday Activities
Benchmarks: Cityscapes ADE20K LVIS Aria Everyday Activities
Metrics: IoU IoU' FLOPs latency
Baselines: Avg+DeepLab Avg+HRNet Avg+SegFormer-B4 Avg+SegFormer-B5 LTD SegFormer-B5
Builds on: Learn-to-Downsample DeepLabV3 SegFormer HRNet U-Net
Closest papers by meaning
- Segment This Thing: Foveated Tokenization for Efficient Point-Prompted Segmentation (CVPR 2025 · similarity 0.85)
- Segment beyond View: Handling Partially Missing Modality for Audio-Visual Semantic Segmentation (AAAI 2024 · similarity 0.85)
- Gaze Beyond the Frame: Forecasting Egocentric 3D Visual Span (NeurIPS 2025 · similarity 0.83)
- Online Segment Any 3D Thing as Instance Tracking (NeurIPS 2025 · similarity 0.83)
- InstanceGaussian: Appearance-Semantic Joint Gaussian Representation for 3D Instance-Level Perception (CVPR 2025 · similarity 0.83)
- Solving Instance Detection from an Open-World Perspective (CVPR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.