Research map / Video / Video segmentation and tracking
Beyond Pixel and Object: Part Feature as Reference for Few-Shot Video Object Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Part Agent Learning Network (PALN), which uses optimal-transport part agents from support images and a cache of temporal part agents from query frames as reference features for few-shot video object segmentation.
Problem
Few-shot video object segmentation needs reliable reference features to align support images with query videos; object prototypes and pixel features are shown to be inadequate for adapting to appearance changes and resisting correlation noise.
Approach
Learn part features as reference via Optimal Transport with an equal-partition constraint to adaptively split support objects into diverse part agents, and use a maximum-orthogonality cache to extract temporal part agents from previous query frames for temporal alignment.
Key results
- On YouTube-FSVOS, PALN reaches 60.9 J (1-shot) and 65.2 J (5-shot), outperforming HPAN and VIPMT by about 3.5 and 1.0 points respectively.
- On MiniVSPW, PALN achieves 32.4 (1-shot) and 34.4 (5-shot) J, with 48 FPS, gains of about 2.0-2.3 points over the best prior FSVOS method.
- Ablation: part agent encoder alone raises 1-shot from 53.9 to 58.1 w/ TAD; the full model reaches 60.9, and part features beat object prototypes and pixel features as reference.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PALN rather than VIPMT | YouTube-FSVOS · 1-shot | J mean: 57.4 -> 60.9 | Helped |
| Use PALN rather than VIPMT | YouTube-FSVOS · 5-shot | J mean: 64.2 -> 65.2 | Helped |
| Use PALN rather than VIPMT | MiniVSPW · 1-shot | J mean: 30.1 -> 32.4 | Helped |
| Use PALN rather than VIPMT | MiniVSPW · 5-shot | J mean: 32.4 -> 34.4 | Helped |
| Add temporal alignment decoder to the pixel-feature SQAD baseline (ablation) | YouTube-FSVOS · 1-shot | J: 53.9 -> 56.4 | Helped |
| Add temporal alignment decoder to the pixel-feature SQAD baseline (ablation) | YouTube-FSVOS · 5-shot | J: 58.7 -> 61.4 | Helped |
| Add PAE to the pixel-feature SQAD baseline (ablation) | YouTube-FSVOS · 1-shot | J: 53.9 -> 58.1 | Helped |
| Add PAE to the pixel-feature SQAD baseline (ablation) | YouTube-FSVOS · 5-shot | J: 58.7 -> 62.2 | Helped |
| Replace object prototypes with learned part features (ablation) | PALN reference features · YouTube-FSVOS | J: 53.3 -> 60.9; 60.4 -> 65.2 | Helped |
| Replace pixel features with learned part features (ablation) | PALN reference features · YouTube-FSVOS | J: 56.4 -> 60.9; 61.4 -> 65.2 | Helped |
| Use learned part agents instead of SLIC or K-Means parts (ablation) | PALN part feature modeling · YouTube-FSVOS | J: 58.8 -> 60.9; 59.7 -> 60.9; 63.2 -> 65.2; 64.9 -> 65.2 | Helped |
| Use the cache mechanism rather than top-Na temporal-agent selection (ablation) | PALN temporal alignment decoder · YouTube-FSVOS | J: 59.0 -> 60.9; 63.8 -> 65.2 | Helped |
Surprising findings
- Top-Na selection of temporal agents performs worse than random sampling, suggesting clustered high-confidence agents lack representativeness.
Limitations
- Relies on pixel-level foreground/background masks and ResNet-50 backbone; generalization to other backbones not reported.
- Evaluated only on two FSVOS benchmarks built from YouTube-VIS and VSPW.
Experimental setup
Datasets: YouTube-FSVOS MiniVSPW
Benchmarks: YouTube-FSVOS MiniVSPW
Metrics: J (region similarity / Jaccard index) FPS
Baselines: PMM PFENet PPNet RePRI NTRENet SSP VAT IPMT DANet TTI HPAN VIPMT
Builds on: DANet HPAN VIPMT PFENet
Closest papers by meaning
- Delving Deep Into Many-to-Many Attention for Few-Shot Video Object Segmentation (CVPR 2021 · similarity 0.90)
- FAPIS: A Few-Shot Anchor-Free Part-Based Instance Segmenter (CVPR 2021 · similarity 0.88)
- PARTONOMY: Large Multimodal Models with Part-Level Visual Understanding (NeurIPS 2025 · similarity 0.86)
- Foreground-Covering Prototype Generation and Matching for SAM-Aided Few-Shot Segmentation (AAAI 2025 · similarity 0.85)
- Learning Spatial-Semantic Features for Robust Video Object Segmentation (ICLR 2025 · similarity 0.85)
- SANSA: Unleashing the Hidden Semantics in SAM2 for Few-Shot Segmentation (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.