Research map / Vision-language and multimodal / Human and embodied modeling
Towards Flexible Visual Relationship Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SwapAnything, a single one-stage transformer that performs standard, promptable, and open-vocabulary visual relationship segmentation (HOI segmentation and panoptic SGG) using SAM-generated masks and CLIP text-matching.
Problem
Visual relationship segmentation has been handled separately for HOI detection, scene graph generation, and referring relationships, with no single model covering standard, prompt-driven, and open-vocabulary settings at pixel level.
Approach
A one-stage Mask2Former-style relationship decoder with latent queries that each output a <subject, predicate, object> triplet (two masks, three class heads), plus textual queries from a frozen CLIP text encoder for structured prompts and for matching class embeddings to open-vocabulary text features; HOI boxes are converted to masks with SAM.
Key results
- On HICO-DET default setting, mask mAP_F of 40.5 (box 38.1) vs. prior single-stage best GEN-VLKT 35.6 mask mAP.
- On promptable VRD, S-IoU 0.568 for subject vs. 0.335 for SSAS.
- Open-vocabulary unseen-object HICO-DET mAP 14.48 unseen / 31.71 full vs. GEN-VLKT 10.51 / 25.63.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use disentangled CE loss instead of triplet CE loss (ablation) | SwapAnything · HICO-DET | mask mAP_F: 36.8 -> 40.5 | Helped |
| Use triplet CE loss instead of disentangled CE loss (ablation) | SwapAnything · V-COCO | AP_role S1: 62.1 -> 66.5 | Helped |
| Use disentangled CE loss instead of triplet CE loss (ablation) | SwapAnything · PSG | R@20: 25.5 -> 27.0 | Helped |
| Use Focal Large instead of Focal Tiny (ablation) | SwapAnything · HICO-DET | mask mAP_F: 34.2 -> 40.5 | Helped |
| Use Focal Large instead of Focal Tiny (ablation) | SwapAnything · V-COCO | AP_role S1: 59.8 -> 66.5 | Helped |
| Use Focal Large instead of Focal Tiny (ablation) | SwapAnything · PSG | R@20: 25.8 -> 27.0 | Helped |
| Add a box head alongside the mask head (ablation) | SwapAnything · HICO-DET | mask mAP_F: 40.5 -> 41.2 | Helped |
| Add a box head alongside the mask head (ablation) | SwapAnything · V-COCO | AP_role S1: 66.5 -> 67.0 | Helped |
| Train on HICO-DET and V-COCO instead of HICO-DET alone (ablation) | SwapAnything · HICO-DET | mask mAP_F: 40.5 -> 40.3 | Hurt |
| Train on HICO-DET and V-COCO instead of HICO-DET alone (ablation) | SwapAnything · V-COCO | AP_role S1: 66.5 -> 66.9 | Helped |
| Train on HICO-DET, V-COCO, and PSG instead of HICO-DET alone (ablation) | SwapAnything · HICO-DET | mask mAP_F: 40.5 -> 40.0 | Hurt |
| Train on HICO-DET, V-COCO, and PSG instead of HICO-DET alone (ablation) | SwapAnything · V-COCO | AP_role S1: 66.5 -> 66.4 | Hurt |
Surprising findings
- Training the existing HOI detectors (CDN, STIP, GEN-VLKT) on SAM-generated masks lowered their box mAP, suggesting the gain is not from SAM data alone.
- Training with boxes only drops the model's mAP (30.7 vs 36.3), attributed to the Mask2Former-derived architecture being designed for pixel-wise segmentation.
Limitations
- Relies on scarce relationship pretraining data and on CLIP, limiting scalability.
- Promptable VRS only supports structured prompts; natural language prompts left for future work.
- No directly comparable baselines for promptable VRS make fair benchmarking hard; lags HiLo on PSG due to long-tail predicates.
Experimental setup
Datasets: HICO-DET V-COCO PSG COCO VRD
Benchmarks: HICO-DET V-COCO PSG VRD
Metrics: mAP role AP R@K mR@K S-IoU O-IoU
Baselines: UniVRD PViC RLIPv2 HOTR QPIC CDN RLIP GEN-VLKT ERNet MUREN PSGTR PSGFormer
Builds on: Mask2Former SAM CLIP GEN-VLKT Focal
Closest papers by meaning
- Multi-Modal Prompting for Open-Vocabulary Video Visual Relationship Detection (AAAI 2024 · similarity 0.87)
- Relation-aware Hierarchical Prompt for Open-vocabulary Scene Graph Generation (AAAI 2025 · similarity 0.87)
- RelationVGGT : Visual Geometry Transformers for 3D Spatial Relation Segmentation (NeurIPS 2026 · similarity 0.85)
- Relationship Prompt Learning is Enough for Open-Vocabulary Semantic Segmentation (NeurIPS 2024 · similarity 0.85)
- Synthetic Visual Genome (CVPR 2025 · similarity 0.85)
- Remodeling Semantic Relationships in Vision-Language Fine-Tuning (AAAI 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.