Research map / Label-efficient and robust learning / Semi-supervised learning
Semi-Supervised Clustering Framework for Fine-grained Scene Graph Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SSC-SGG, a semi-supervised clustering framework that uses prototype-based multi-view clustering and a dynamic mini-batch pseudo-label assignment to mine low-frequency interaction samples from unlabeled object pairs for scene graph generation.
Problem
Scene graph annotations are sparse and biased, so unlabeled object pairs are treated as background, causing confusion training and biased predictions against low-frequency predicates.
Approach
Use labeled-data-initialized prototypes to cluster multi-view relationship features with swap-prediction, then assign soft pseudo-labels to unlabeled pairs via Sinkhorn-based optimal transport with weights adjusted by historical assignment counts, so low-frequency predicates get balanced assignments.
Key results
- On VG150 with Motifs, SSC-SGG reaches PredCls M@100 of 48.0 vs 42.3 for vanilla Motifs and 47.4 for CFA.
- Compared with pseudo-labeling methods IETrans and ST-SGG, SSC-SGG gains about 10.2% and 11.8% average M@100 across subtasks.
- On Open Image V6 SGDet with Transformer, the MPC+SSC-SGG variant reaches mR@50 of 44.26 vs 32.12 for baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add SSC-SGG to Motifs | Motifs · VG150 PredCls | M@100: 42.3 -> 48.0 | Helped |
| add SSC-SGG to Motifs | Motifs · VG150 SGCls | M@100: 24.2 -> 28.7 | Helped |
| add SSC-SGG to Motifs | Motifs · VG150 SGDet | M@100: 21.9 -> 23.7 | Helped |
| add SSC-SGG to Motifs | Motifs · Open Image V6 SGDet | mR@50: 32.68 -> 42.48 | Helped |
| add multi-view prototype-based clustering without pseudo-label assignment (ablation) | Motifs · VG150 PredCls | M@100: 42.3 -> 45.0 | Helped |
| add dynamic pseudo-label assignment to MPC (ablation) | Motifs with MPC · VG150 PredCls | M@100: 45.0 -> 48.0 | Helped |
| remove multi-view prototype-based clustering (ablation) | Motifs · VG150 PredCls | M@100: 45.0 -> 42.3 | Hurt |
| remove dynamic pseudo-label assignment (ablation) | Motifs with MPC · VG150 PredCls | M@100: 48.0 -> 45.0 | Hurt |
| use static hard pseudo-labels instead of dynamic soft pseudo-labels (ablation) | Motifs · VG150 PredCls | M@100: 48.0 -> 45.3 | Hurt |
| use static soft pseudo-labels instead of dynamic soft pseudo-labels (ablation) | Motifs · VG150 PredCls | M@100: 48.0 -> 47.2 | Hurt |
| use static hard pseudo-labels instead of no assignment (ablation) | Motifs · VG150 PredCls | M@100: 45.0 -> 45.3 | No clear effect |
| replace multi-view relationship augmentation with a single view (ablation) | SSC-SGG · VG150 PredCls | M@100: 48.0 -> 47.0 | Hurt |
Surprising findings
- Multi-view clustering alone gives almost no gain without pseudo-labels; its benefit appears only in the semi-supervised setting.
- SSC-SGG drops R@50 relative to baseline in PredCls (59.7 vs 65.2) while raising mR@50 substantially.
Limitations
- Pseudo-labeling only top-5 samples per image on average, with hyperparameters (lambda, t, sigma, n) manually set
- Multi-view clustering gives little benefit without the semi-supervised pseudo-labels
- Detector parameters are frozen, so proposal quality is not improved
Experimental setup
Datasets: Visual Genome VG150 Open Image V6
Benchmarks: Visual Genome VG150 Open Image V6
Metrics: R@K mR@K M@K wmAP scorewtd
Baselines: TDE NICE IETrans CFA ST-SGG CogTree RelDN RU-Net PE-Net SGTR+
Builds on: Sinkhorn-Knopp optimal transport SwAV swap prediction Motifs IETrans PE-Net prototype regularization
Closest papers by meaning
- Adaptive Self-training Framework for Fine-grained Scene Graph Generation (ICLR 2024 · similarity 0.89)
- RA-SGG: Retrieval-Augmented Scene Graph Generation Framework via Multi-Prototype Learning (AAAI 2025 · similarity 0.89)
- Scene Graph Generation Strategy with Co-occurrence Knowledge and Learnable Term Frequency (ICML 2024 · similarity 0.88)
- Scene Graph-Grounded Image Generation (AAAI 2025 · similarity 0.87)
- TD²-Net: Toward Denoising and Debiasing for Video Scene Graph Generation (AAAI 2024 · similarity 0.87)
- DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.