Research map / Video / Video segmentation and tracking
Motion-aware Contrastive Learning for Temporal Panoptic Scene Graph Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a motion-aware contrastive learning framework for temporal panoptic scene graph generation that aligns mask tubes with optimal transport distance and uses shuffled-tube and same-video triplet negatives.
Problem
Temporal panoptic scene graph generation models pool mask tubes over time before relation classification, which discards motion cues needed to recognize dynamic relations, and frame-level video contrastive learning does not target entity-level motion.
Approach
Contrastive learning over entity mask tubes: positives are tubes of similar subject-relation-object triplets from other videos, negatives are temporally shuffled tubes and tubes of different triplets from the same video, with similarity computed by Sinkhorn optimal transport distance over per-frame features rather than pooled features.
Key results
- On OpenPVSG with IPS+T at vIoU 0.5, R@50 reaches 6.08 vs 5.24 for IPS+T - Convolution.
- On PSG4D-HOI with point cloud videos, R@50 improves from 5.61 (PSG4DFormer) to 7.62.
- On PSG4D-GTA with RGB-D videos, R@20 improves from 6.68 (PSG4DFormer) to 9.07.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add motion-aware contrastive learning | IPS+T - Convolution · OpenPVSG, vIoU 0.5 | R/mR@50: 5.24 / 3.29 -> 6.08 / 4.38 | Helped |
| add motion-aware contrastive learning | PSG4DFormer, point cloud videos · PSG4D-GTA | R/mR@50: 4.83 / 2.93 -> 6.31 / 3.70 | Helped |
| add motion-aware contrastive learning | PSG4DFormer, RGB-D videos · PSG4D-HOI | R/mR@50: 6.16 / 4.16 -> 8.36 / 6.94 | Helped |
| add shuffle-based contrastive learning (ablation) | Ours - Convolution · OpenPVSG, vIoU 0.5 | R/mR@50: 6.02 / 4.28 -> 6.08 / 4.38 | Helped |
| add triplet-based contrastive learning (ablation) | Ours - Convolution · OpenPVSG, vIoU 0.5 | R/mR@50: 5.90 / 4.24 -> 6.08 / 4.38 | Helped |
| add shuffle-based contrastive learning (ablation) | Ours · PSG4D-GTA, point cloud videos | R/mR@50: 5.59 / 3.26 -> 6.31 / 3.70 | Helped |
| add triplet-based contrastive learning (ablation) | Ours · PSG4D-GTA, point cloud videos | R/mR@50: 5.57 / 2.98 -> 6.31 / 3.70 | Helped |
| use optimal transport instead of temporal pooling with cosine similarity (ablation) | Ours - Convolution · OpenPVSG, vIoU 0.5 | R/mR@50: 6.04 / 4.36 -> 6.08 / 4.38 | Helped |
| use optimal transport instead of temporal pooling with L2 distance (ablation) | Ours - Convolution · OpenPVSG, vIoU 0.5 | R/mR@50: 3.77 / 5.95 -> 6.08 / 4.38 | Mixed |
| use optimal transport instead of temporal pooling with cosine similarity (ablation) | Ours · PSG4D-GTA, point cloud videos | R/mR@50: 6.02 / 3.62 -> 6.31 / 3.70 | Helped |
| use optimal transport instead of temporal pooling with L2 distance (ablation) | Ours · PSG4D-GTA, point cloud videos | R/mR@50: 5.38 / 3.39 -> 6.31 / 3.70 | Helped |
| increase the threshold for selecting strong-motion tubes (ablation) | Ours · OpenPVSG | performance | Mixed |
Surprising findings
- Triplet-based negatives contribute more than shuffle-based negatives, which the authors attribute to shuffle-based being better at focusing on motion.
Limitations
- Static relations with almost no motion make shuffled tubes identical to anchors, requiring a motion-threshold selection step (gamma) tuned on validation.
- Gains on some metrics are small (often around 1 point), and the paper reports mixed results on mR for some settings.
Experimental setup
Datasets: OpenPVSG PSG4D-GTA PSG4D-HOI
Benchmarks: OpenPVSG PSG4D-GTA PSG4D-HOI
Metrics: R@K mR@K vIoU
Baselines: IPS+T - Vanilla IPS+T - Handcrafted filter IPS+T - Convolution IPS+T - Transformer VPS - Vanilla VPS - Handcrafted filter VPS - Convolution VPS - Transformer 3DSGG PSG4DFormer
Builds on: Mask2Former UniTrack Video K-Net PSG4DFormer Sinkhorn algorithm
Closest papers by meaning
- R3CD: Scene Graph to Image Generation with Relation-Aware Compositional Contrastive Control Diffusion (AAAI 2024 · similarity 0.87)
- Towards Unbiased and Robust Spatio-Temporal Scene Graph Generation and Anticipation (CVPR 2025 · similarity 0.87)
- DiffVsgg: Diffusion-Driven Online Video Scene Graph Generation (CVPR 2025 · similarity 0.86)
- Through-The-Mask: Mask-based Motion Trajectories for Image-to-Video Generation (CVPR 2025 · similarity 0.85)
- Learning Temporal 3D Semantic Scene Completion via Optical Flow Guidance (NeurIPS 2025 · similarity 0.85)
- Object-Centric Representation Learning for Enhanced 3D Semantic Scene Graph Prediction (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.