Research map / Graph learning / Knowledge and temporal graphs
Scene Graph Generation Strategy with Co-occurrence Knowledge and Learnable Term Frequency
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CooK (object co-occurrence knowledge injected into MPNN message passing) and a learnable TF-l-IDF layer to improve MPNN-based scene graph generation under long-tail distributions.
Problem
Scene graph generation with MPNN-based models ignores object co-occurrence knowledge and suffers from long-tail class distributions in relation and object labels.
Approach
Compute an object-to-object co-occurrence probability matrix (CooK) from training data and use it to weight MPNN messages, plus a batch-level learnable TF-l-IDF layer (with trainable epsilon and gamma in the log term) that rescales node features to boost tail classes and suppress head classes.
Key results
- On VG SGGen, CooK + TF-l-IDF reaches mR@50 14.2 (vs. 12.2 for Nice-Motif/HetSGG), a claimed gain of up to 3.8% over SoTA.
- On Open Images, CooK + TF-l-IDF achieves score_wtd 45.1 vs. 44.9 for PE-Net.
- Applied to GPS-Net, SGGen mR@50 improves from 6.7 to 8.3, while on BGNN R@50 drops from 31.0 to 29.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CooK and TF-l-IDF to the SGG method | Visual Genome SGG · Visual Genome PredCls | mR@50: 33.7 / 35.8 -> 35.4 / 37.2 | Helped |
| Add CooK and TF-l-IDF to the SGG method | Visual Genome SGG · Visual Genome SGCls | mR@50: 17.5/ 18.6 -> 19.1 / 20.3 | Helped |
| Add CooK and TF-l-IDF to the SGG method | Visual Genome SGG · Visual Genome SGGen | mR@50: 14.1 / 16.5 -> 14.2 / 16.3 | Mixed |
| Add CooK and TF-l-IDF | Open Images SGG · Open Images SGGen | score_wtd: 44.9 -> 45.1 | Helped |
| Add CooK and TF-l-IDF to G-RCNN | G-RCNN · Visual Genome SGGen | mR@50: 5.8 -> 7.1 | Helped |
| Add CooK and TF-l-IDF to GPS-Net | GPS-Net · Visual Genome SGGen | mR@50: 6.7 -> 8.3 | Helped |
| Add CooK and TF-l-IDF to BGNN | BGNN · Visual Genome SGGen | mR@50: 10.7 -> 11.4 | Helped |
| Add learnable parameters to TF-l-IDF (ablation) | TF-l-IDF · Visual Genome PredCls | mR@50: 31.6 -> 33.6 | Helped |
| Add learnable parameters to TF-l-IDF (ablation) | TF-l-IDF · Visual Genome SGCls | mR@50: 15.1 -> 18.5 | Helped |
| Add learnable parameters to TF-l-IDF (ablation) | TF-l-IDF · Visual Genome SGGen | mR@50: 11.1 -> 12.8 | Helped |
| Replace VG-only CooK with advanced CooK from VG and Open Images (ablation) | CooK + TF-l-IDF · Visual Genome SGGen | mR@50: 14.2 -> 14.9 | Helped |
| Increase batch size for TF-l-IDF (ablation) | TF-l-IDF · Visual Genome PredCls, SGCls, and SGGen | performance | Helped |
Surprising findings
- Applying the method to BGNN lowers SGGen R@50 (31.0 to 29.8) and R@100 vs. baseline, and CooK+TF-l-IDF has lower R@K than CooK alone in PredCls and SGGen on VG.
- Learnable epsilon and gamma in the l-IDF term give an average 2.4% gain over the non-learnable version.
Limitations
- Only applicable to MPNN-based SGG models, not non-MPNN architectures.
- CooK learning is limited to supervised learning and hard to apply to self-supervised foundation models.
- TF-l-IDF depends on batch size, so performance improves with larger batches.
Experimental setup
Datasets: Visual Genome Open Images
Benchmarks: Visual Genome Open Images
Metrics: mR@K R@K wmAP_rel wmAP_phr score_wtd
Baselines: IMP KERN Motifs VCTree GPS-Net BGNN HetSGG PE-Net VETO+Rwt Nice-Motif RU-Net SQUAT
Builds on: Graph-R-CNN TF-IDF PySGG
Closest papers by meaning
- Adaptive Visual Scene Understanding: Incremental Scene Graph Generation (NeurIPS 2024 · similarity 0.89)
- Scene Graph-Grounded Image Generation (AAAI 2025 · similarity 0.89)
- DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation (CVPR 2024 · similarity 0.88)
- Semi-Supervised Clustering Framework for Fine-grained Scene Graph Generation (AAAI 2025 · similarity 0.88)
- From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models (CVPR 2024 · similarity 0.88)
- RA-SGG: Retrieval-Augmented Scene Graph Generation Framework via Multi-Prototype Learning (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.