Research map / Detection and segmentation / CNN features
MobileInst: Video Instance Segmentation on the Mobile
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MobileInst, a lightweight query-based video instance segmentation framework for mobile devices that tracks objects via kernel reuse and cosine-similarity kernel association, achieving 35.0 AP on YouTube-VIS 2019 at 188 ms on a Snapdragon 778G CPU core.
Problem
Video instance segmentation on resource-constrained mobile devices is hard because clip-level models are too heavy and frame-level models rely on heuristic tracking such as NMS-based association.
Approach
A dual transformer instance decoder (global and local decoders) with a semantic-enhanced mask decoder, plus tracking by reusing object-query mask kernels across short clips and associating kernels by cosine similarity, trained with temporal query passing; no extra tracking modules or heuristics are needed.
Key results
- On COCO test-dev, MobileInst with TopFormer at 640 reaches 31.2 mask AP at 433 ms on one Snapdragon 778G CPU core, about 50% lower latency than SparseInst.
- On YouTube-VIS 2019, MobileInst achieves 35.0 AP at 188 ms on mobile, versus 33.3 AP at 389 ms for SparseInst-VIS (TopFormer).
- On YouTube-VIS 2021, MobileInst achieves 30.1 AP; temporal training improves AP by 1.3 for the global-local decoder at T=1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MobileInst with TopFormer rather than SparseInst-VIS | YouTube-VIS 2019 video instance segmentation · YouTube-VIS 2019 val | AP: 33.3 -> 35.0 | Helped |
| Use MobileInst with TopFormer rather than SparseInst-VIS | YouTube-VIS 2019 video instance segmentation · YouTube-VIS 2019 val | mobile latency: 389 -> 188 | Hurt |
| Add the local instance decoder to a global-only decoder (ablation) | MobileInst instance decoder · COCO val2017 | AP: 28.7 -> 29.8 | Helped |
| Add the global instance decoder to a local-only decoder (ablation) | MobileInst instance decoder · COCO val2017 | AP: 29.3 -> 29.8 | Helped |
| Replace two local decoders with the global-local decoder (ablation) | MobileInst video instance decoder · YouTube-VIS 2021 val | AP: 28.5 -> 29.3 | Helped |
| Use no pooling instead of 8x8 max pooling for local features (ablation) | MobileInst local instance decoder · COCO val2017 | AP: 29.8 -> 30.7 | Helped |
| Use 8x8 max pooling instead of 8x8 average pooling (ablation) | MobileInst local instance decoder · COCO val2017 | AP: +0.4 AP | Helped |
| Add temporal query-passing training (ablation) | MobileInst global-local decoder · YouTube-VIS 2021 val | AP: 28.0 -> 29.3 | Helped |
| Add temporal query-passing training (ablation) | MobileInst local-local decoder · YouTube-VIS 2021 val | AP: 27.8 -> 28.6 | Helped |
| Replace 8x8 max pooling with no pooling for local features (ablation) | MobileInst local instance decoder · COCO val2017 | latency: 427ms -> 613ms | Hurt |
| Use the proposed iterative multi-scale mask decoder instead of FPN with 1x conv (ablation) | MobileInst mask decoder · COCO val2017 | AP: 28.7 -> 29.8 | Helped |
| Add the semantic enhancer to the mask decoder (ablation) | MobileInst mask decoder · COCO val2017 | AP: 29.8 -> 30.1 | Helped |
Surprising findings
- Reusing mask kernels across frames (T=3) improved speed without performance loss in short clips, and temporal query passing gave gains at T=3 with the global-local decoder.
- Stacking two local decoders matched the global-local decoder on COCO (29.8 AP) but was worse for video tracking.
Limitations
- Kernel reuse tends to fail in long sequences or with drastic frame changes, motivating kernel association.
- Coarse mask boundaries under severe motion.
Experimental setup
Datasets: COCO train2017 YouTube-VIS 2019 YouTube-VIS 2021
Benchmarks: COCO test-dev COCO val2017 YouTube-VIS 2019 YouTube-VIS 2021
Metrics: mask AP AP_S AR1 latency (ms)
Baselines: Mask R-CNN CondInst SOLOv2-Lite YOLACT SparseInst Mask2Former FastInst CrossVIS Mask Track R-CNN SipMask SGMask STMask
Builds on: Mask2Former SparseInst IDOL TopFormer SeaFormer
Closest papers by meaning
- FastInst: A Simple Query-Based Model for Real-Time Instance Segmentation (CVPR 2023 · similarity 0.88)
- InstanceFormer: An Online Video Instance Segmentation Framework (AAAI 2023 · similarity 0.87)
- SOLOv2: Dynamic and Fast Instance Segmentation (NeurIPS 2020 · similarity 0.87)
- VideoCutLER: Surprisingly Simple Unsupervised Video Instance Segmentation (CVPR 2024 · similarity 0.84)
- Robust and Consistent Online Video Instance Segmentation via Instance Mask Propagation (AAAI 2025 · similarity 0.84)
- Minimizing Labeled, Maximizing Unlabeled: An Image-Driven Approach for Video Instance Segmentation (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.