Research map / Label-efficient and robust learning / Recognition and long-tail
Multi-Prototype Space Learning for Commonsense-Based Scene Graph Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes multi-prototype learning (MPL) for commonsense-based scene graph generation, matching each predicate to one of K prototypes per class via triple-level optimal transport, with momentum prototype updates and separability/compactness losses.
Problem
Commonsense-based scene graph generation uses a single prototype per predicate class, which cannot capture large intra-class visual variation, causing diverse predicates (e.g., riding, standing on, lying on) to be misclassified.
Approach
Maintain multiple prototypes per predicate class, learned by matching predicate features to prototypes with a triple-level optimal transport that also uses object-predicate and subject-predicate distances, updated by momentum and optimized with inter-class separability and intra-class compactness losses.
Key results
- On Visual Genome, the method reaches a mean recall of 54.2 across SGDET, SGCLS, and PREDCLS, outperforming listed state-of-the-art methods such as PE-Net (46.4) and KEM (52.3).
- On Open Images V6, mR@50 is 43.98 versus 42.43 for BGNN+SCR, and R@50 is 76.34 versus 75.21.
- Ablation on VG shows K=3 prototypes per class gives the best SGDET mR@50 of 21.2; K=1 gives 19.7 and K=5 gives 19.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multi-prototype learning instead of fixed single-prototype commonsense (steps 1 and 2) (ablation) | MPL · Visual Genome SGDET | mR@50: 18.8 -> 21.2 | Helped |
| Replace traditional OT with triple-level optimal transport (TOT) (ablation) | MPL · Visual Genome SGDET | mR@50: 19.1 -> 21.2 | Helped |
| Add inter-class separability and intra-class compactness losses (LICS and LICC) (ablation) | MPL · Visual Genome SGDET | mR@50: 19.7 -> 21.2 | Helped |
| Add inter-class separability loss (LICS) (ablation) | MPL · Visual Genome SGDET | mR@50: 20.9 -> 21.2 | Helped |
| Add intra-class compactness loss (LICC) (ablation) | MPL · Visual Genome SGDET | mR@50: 20.7 -> 21.2 | Helped |
| Set prototype number K=1 instead of K=3 (ablation) | MPL · Visual Genome SGDET | mR@50: 21.2 -> 19.7 | Hurt |
| Set prototype number K=2 instead of K=3 (ablation) | MPL · Visual Genome SGDET | mR@50: 21.2 -> 20.1 | Hurt |
| Set prototype number K=4 instead of K=3 (ablation) | MPL · Visual Genome SGDET | mR@50: 21.2 -> 19.9 | Hurt |
| Set prototype number K=5 instead of K=3 (ablation) | MPL · Visual Genome SGDET | mR@50: 21.2 -> 19.3 | Hurt |
Surprising findings
- Increasing the prototype count beyond 3 degrades performance, attributed to noisy messages in the optimal transport.
Limitations
- Gains depend on a pre-trained GCN commonsense graph from GB-Net for initial prototypes.
- Many hyperparameters (K, beta, epsilon, lambda, tau, delta) are tuned by cross-validation or fixed by hand.
Experimental setup
Datasets: Visual Genome Open Images V6
Benchmarks: Visual Genome Open Images V6
Metrics: R@20 R@50 R@100 mR@50 mR@100 wmAP rel wmAP phr score wtd
Baselines: BGNN NICE SMP GB-Net Motif + DNS TXM KEM V S3 PE-Net RelDN VCTree MOTIFS
Builds on: GB-Net Sinkhorn-Knopp Focal loss Pseudo-Huber loss
Closest papers by meaning
- LLM4SGG: Large Language Models for Weakly Supervised Scene Graph Generation (CVPR 2024 · similarity 0.87)
- Leveraging Predicate and Triplet Learning for Scene Graph Generation (CVPR 2024 · similarity 0.87)
- Navigating the Unseen: Zero-shot Scene Graph Generation via Capsule-Based Equivariant Features (CVPR 2025 · similarity 0.87)
- Synthetic Visual Genome (CVPR 2025 · similarity 0.87)
- Panoptic Scene Graph Generation with Semantics-Prototype Learning (AAAI 2024 · similarity 0.87)
- Scene Graph Generation with Role-Playing Large Language Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.