Research map / Video / Video segmentation and tracking

GMOS: Grounding Moving Object Segmentation in 3D Space and Time

NeurIPS 2026 · Video › Video segmentation and tracking · approach: Matching-based video object segmentation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GMOS, a proposer-propagator framework that performs 3D-aware, instantaneous moving object segmentation from raw RGB video using frozen pi3 and SAM2 encoders, plus a new temporally fine-grained dataset GMOS-2K and the MOS-I protocol.

Problem

Existing moving object segmentation methods rely on pre-computed 2D auxiliary signals (optical flow, point tracks) lacking 3D geometry and treat motion as a sequence-level attribute, ignoring per-frame motion state.

Approach

Combine frozen feed-forward 3D reconstruction features (pi3) with SAM2 segmentation features in a query-based proposer that jointly predicts masks and per-frame motion states, linked into tracks by a SAM2 propagator; add a confidence-modulated loss that down-weights noisy static-moving transition labels.

Key results

• GMOS reports 79.7 J / 86.0 F on DAVIS17-M and 82.0 J on SegTrackv2 and 0.837 SR on MoCA, improving over prior SoTA on the multi-object and foreground-background MOS benchmarks.
• On MOS-I, GMOS reaches 72.9 mtIoU on DAVIS17-IM versus 67.0 for SegAnyMo, with J_mov 84.2 and FP count 0.079.
• GMOS runs at 0.38 s per frame versus 4.06 s for SegAnyMo, reported as roughly three times faster than prior MOS methods (GMOS-S: 0.26 s).

Surprising findings

• The online variant performs on par with the offline mode on MOS-I, and the offline second pass slightly hurts FP count and mtIoU.
• Using pi3 features alone without SAM2 (segmentation encoder removed) yields the lowest mtIoU in the proposer ablation, indicating neither encoder alone suffices.

Limitations

• Relies on frozen pi3 and SAM2 encoders, so performance depends on these backbones.
• Online mode is restricted to MOS-I because it cannot recover masks for objects that are initially static and later move.
• Training data for MOS-I relies on temporal motion annotations of limited size (1,930 real training videos).

Experimental setup

Datasets: Kubric PointOdyssey DynamicReplica DAVIS17 YTVOS19 OVIS MoCA-Mask HOI4D Mannequin Challenge SegTrackv2 FBMS-59 DAVIS16

Benchmarks: YTVOS19-M DAVIS17-M DAVIS16-M SegTrackv2 FBMS-59 MoCA YTVOS19-IM DAVIS17-IM DAVIS16 DAVIS17

Metrics: J (region similarity) F (contour accuracy) SR J_mov FP count mtIoU

Baselines: FlowSAM SegAnyMo GeoMotion OCLR-TTA ABR DEVA

Builds on: SAM2 pi3 FlowSAM SegAnyMo GeoMotion

Closest papers by meaning

• Segmenting Moving Objects via an Object-Centric Layered Representation (NeurIPS 2022 · similarity 0.89)
• Long-RVOS: A Comprehensive Benchmark for Long-term Referring Video Object Segmentation (CVPR 2026 · similarity 0.88)
• M^3-VOS: Multi-Phase, Multi-Transition, and Multi-Scenery Video Object Segmentation (CVPR 2025 · similarity 0.88)
• Segment Any Motion in Videos (CVPR 2025 · similarity 0.88)
• Discovering Objects That Can Move (CVPR 2022 · similarity 0.88)
• Learning Segmentation from Point Trajectories (NeurIPS 2024 · similarity 0.87)