Research map / Vision-language and multimodal / Human and embodied modeling
Leveraging Predicate and Triplet Learning for Scene Graph Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Dual-granularity Relation Modeling (DRM), which models coarse predicate cues and fine-grained triplet cues with contrastive constraints, plus Dual-granularity Knowledge Transfer (DKT) that transfers head-class Gaussian variance to tail classes for scene graph generation.
Problem
Scene graph generation suffers from large visual variation within the same predicate across different subject-object pairs and from the long-tail distribution, where tail predicates have few patterns and triplet types.
Approach
Model predicate representations jointly with fine-grained triplet representations (subject, predicate, object) under dual-granularity contrastive constraints, and transfer covariance from head predicates/triplets to tail ones via Gaussian sampling to enrich tail patterns.
Key results
- On VG150 PredCls, DRM w/o DKT outperforms PE-Net by 2.0% at R@100, and DRM outperforms SHA+GCL by 5.5% at mR@100 on PredCls.
- On Open Image V6, DRM w/o DKT reaches scorewtd 40.5 versus 36.6 for PE-Net (the strongest listed baseline on that score).
- On GQA200, DRM outperforms SHA+GCL by an average of 0.4% at mR@100 across three tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add predicate and triplet cue modeling (ablation) | DRM · VG150 PredCls | R@50: 56.5/60.4 -> 69.3/71.3 | Helped |
| Add dual-granularity constraints loss (ablation) | DRM · VG150 PredCls | R@50/100: 69.7/71.5 -> 70.2/72.1 | Helped |
| Add dual-granularity augmentation (ablation) | DRM · VG150 PredCls | mR@50/100: 20.8/22.8 -> 22.0/24.0 | Helped |
| Transfer predicate knowledge only (ablation) | DRM · VG150 PredCls | mR@50/100: 23.3/25.6 -> 45.0/47.3 | Helped |
| Transfer triplet knowledge only (ablation) | DRM · VG150 PredCls | mR@50/100: 23.3/25.6 -> 46.1/48.7 | Helped |
| Transfer predicate and triplet knowledge (ablation) | DRM · VG150 PredCls | mR@50/100: 23.3/25.6 -> 47.1/49.6 | Helped |
| Add DKT (ablation) | DRM · VG150 PredCls | R@50: 70.2 -> 43.9 | Hurt |
| Add DKT (ablation) | DRM · GQA200 PredCls | mR@100: 19.0 -> 43.5 | Helped |
| Use DRM w/o DKT instead of PE-Net | Visual Genome SGG · VG150 PredCls | R@100: 70.1 -> 72.1 | Helped |
| Use DRM w/o DKT instead of PE-Net | Open Image V6 SGG · Open Image V6 | WmAPrel: 36.6 -> 40.5 | Helped |
Surprising findings
- DKT reduces R@K on head predicates (e.g., 'on') because ambiguous predicates are reclassified into specific tail predicates, which the authors note is inevitable trade-off.
Limitations
- R@K drops for head predicates when DKT is applied, trading off frequent-predicate recall.
- Relies on a pre-trained Faster R-CNN proposal network, so SGDet performance depends on the detector.
Experimental setup
Datasets: Visual Genome VG150 Open Image V6 GQA200
Benchmarks: Visual Genome VG150 Open Image V6 GQA200
Metrics: Recall@K mean Recall@K weighted mAP weighted score
Baselines: MOTIFS VCTREE PE-Net VETO SHA+GCL BGNN SQUAT CaCao NICE EICR TDE IETrans
Builds on: PE-Net SHA / GCL Faster R-CNN GloVe MOTIFS
Closest papers by meaning
- LLM4SGG: Large Language Models for Weakly Supervised Scene Graph Generation (CVPR 2024 · similarity 0.90)
- RA-SGG: Retrieval-Augmented Scene Graph Generation Framework via Multi-Prototype Learning (AAAI 2025 · similarity 0.89)
- DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation (CVPR 2024 · similarity 0.89)
- From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models (CVPR 2024 · similarity 0.89)
- Hydra-SGG: Hybrid Relation Assignment for One-stage Scene Graph Generation (ICLR 2025 · similarity 0.89)
- Adaptive Self-training Framework for Fine-grained Scene Graph Generation (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.