Research map / Graph learning / Knowledge and temporal graphs
Panoptic Scene Graph Generation with Semantics-Prototype Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ADTrans, a plug-and-play framework that uses prototype-based predicate representation learning to adaptively transfer biased predicate annotations (indistinguishable triplets and NA samples) into more consistent labels for panoptic/scene graph generation.
Problem
Biased predicate annotations in PSG and SGG datasets (semantic overlap between predicates and annotator preferences) give different predicates for the same object pairs, hurting model decision boundaries and long-tail performance.
Approach
Fine-tune BERT with a robust contrastive objective (angular margin, visual-confusion weighting, loss-variance invariance regularization) to learn predicate prototypes, then use prototype similarity as adaptive transfer ratios and influence factors to relabel biased and potentially positive samples, plus a repeat-factor resampling.
Key results
- On PSG SGDet with PSGTR, ADTrans raises mR@100 from 22.1 to 30.0 and PR@100 from 27.6 to 30.5 (vs IETrans 28.7).
- On VG PREDCLS, MOTIFS mR@100 improves from 16.2 to 38.8 with ADTrans.
- Ablation: InfoNCE gives 26.3 mR@100 on SGDet versus 30.0 with the proposed robust contrastive training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ADTrans to PSGTR | PSGTR · PSG SGDet | mR@100: 22.1 -> 30.0 | Helped |
| Add ADTrans to MOTIFS | MOTIFS · VG PREDCLS | mR@100: 16.2 -> 38.8 | Helped |
| Add ADTrans to VCTree | VCTree · VG SGCLS | mR@100: 8.0 -> 19.8 | Helped |
| Add ADTrans to GPSnet | GPSnet · VG SGDET | mR@100: 7.1 -> 19.2 | Helped |
| Add indistinguishable triplet transfer (ablation) | PSGTR · PSG SGDet | mR@100: 22.1 -> 29.2 | Helped |
| Add potentially positive transfer after indistinguishable triplet transfer (ablation) | PSGTR + IT · PSG SGDet | mR@100: 29.2 -> 29.6 | Helped |
| Add re-sampling to IT and PT (ablation) | PSGTR + IT + PT · PSG SGDet | mR@100: 29.6 -> 30.0 | Helped |
| Remove indistinguishable triplet pairs instead of transferring them (ablation) | PSGTR · PSG SGDet | mR@100: 22.1 -> 25.3 | Helped |
| Transfer indistinguishable triplet pairs rather than simply removing them (ablation) | PSGTR · PSG SGDet | mR@100: 25.3 -> 29.2 | Helped |
| Use InfoNCE contrastive learning (ablation) | PSGTR · PSG SGDet | mR@100: 22.1 -> 26.3 | Helped |
| Use robust contrastive training instead of InfoNCE (ablation) | PSGTR · PSG SGDet | mR@100: 26.3 -> 30.0 | Helped |
Surprising findings
- Simply removing indistinguishable triplet pairs greatly outperforms the original baseline (mR@20 from 16.6 to 20.0), indicating harm from biased annotations.
Limitations
- Relies on a pre-trained VCTree to identify indistinguishable and NA samples, so transfer quality depends on that model.
- Many hyperparameters (margin, beta, gamma, D, K_g, t) are set without reported sensitivity analysis.
Experimental setup
Datasets: PSG dataset Visual Genome
Benchmarks: PSG dataset Visual Genome
Metrics: R@K mR@K PR@K F@K
Baselines: IMP IETrans VCTree MOTIFS GPSnet PSGTR
Builds on: IETrans PSGTR VCTree InfoNCE
Closest papers by meaning
- Noise-Guided Predicate Representation Extraction and Diffusion-Enhanced Discretization for Scene Graph Generation (ICML 2025 · similarity 0.89)
- From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models (CVPR 2024 · similarity 0.88)
- Leveraging Predicate and Triplet Learning for Scene Graph Generation (CVPR 2024 · similarity 0.88)
- Synthetic Visual Genome (CVPR 2025 · similarity 0.87)
- DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation (CVPR 2024 · similarity 0.87)
- RA-SGG: Retrieval-Augmented Scene Graph Generation Framework via Multi-Prototype Learning (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.