Research map / Graph learning / Knowledge and temporal graphs
Adaptive Self-training Framework for Fine-grained Scene Graph Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ST-SGG, a self-training framework for scene graph generation that pseudo-labels unannotated triplets using class-specific adaptive thresholds with momentum (CATM) and a graph structure learner for MPNN-based models.
Problem
Scene graph generation suffers from long-tailed predicate distributions and missing annotations (unannotated triplets labeled as background), which hurts fine-grained predicate prediction.
Approach
Self-training for SGG where pseudo-labels for unannotated triplets are assigned per batch using class-specific thresholds updated by EMA, with momentum rates scaled by predicate class frequency so tail classes are pseudo-labeled more actively; a graph structure learner (GSL) enriches the scene graph for MPNN-based models.
Key results
- Motif+I-Trans+ST-SGG reaches F@50/100 of 41.7/42.5 on PredCls on Visual Genome, versus 38.1/40.9 for Motif+I-Trans.
- BGNN+ST-SGG+GSL improves SGCls F@100 to 24.9 versus 22.5 for BGNN+Resampling on Visual Genome.
- Fixed-threshold self-training baselines (constant, class-specific, frequency-weighted) underperform vanilla Motif across all metrics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ST-SGG to Motif | Motif · Visual Genome | mR@K: 17.8 / 19.2 -> 22.4 / 24.1 (PredCls); 9.0 / 9.6 -> 12.1 / 12.8 (SGCls); 6.4 / 7.6 -> 8.5 / 10.1 (SGDet) | Helped |
| Add ST-SGG to VCTree | VCTree · Visual Genome | mR@K: 17.2 / 18.6 -> 21.5 / 22.9 (PredCls); 9.6 / 10.2 -> 12.0 / 12.5 (SGCls); 7.3 / 8.6 -> 8.7 / 10.1 (SGDet) | Helped |
| Add ST-SGG to BGNN with resampling | BGNN + resampling · Visual Genome | mR@K: 29.2 / 31.7 -> 33.0 / 35.1 (PredCls); 14.6 / 16.0 -> 17.9 / 19.0 (SGCls); 11.4 / 13.3 -> 13.6 / 16.1 (SGDet) | Helped |
| Add GSL to BGNN + ST-SGG with resampling | BGNN + resampling + ST-SGG · Visual Genome | mR@K: 33.0 / 35.1 -> 34.1 / 36.2 (PredCls); 17.9 / 19.0 -> 18.0 / 19.4 (SGCls); 13.6 / 16.1 -> 14.1 / 16.6 (SGDet) | Helped |
| Add ST-SGG to HetSGG with resampling | HetSGG + resampling · Visual Genome | mR@K: 30.0 / 32.2 -> 32.6 / 35.2 (PredCls); 15.8 / 17.7 -> 18.0 / 18.9 (SGCls); 11.5 / 13.5 -> 11.9 / 14.6 (SGDet) | Helped |
| Add GSL to HetSGG + ST-SGG with resampling | HetSGG + resampling + ST-SGG · Visual Genome | mR@K: 32.6 / 35.2 -> 33.6 / 35.8 (PredCls); 18.0 / 18.9 -> 18.2 / 19.1 (SGCls); 11.9 / 14.6 -> 12.9 / 15.0 (SGDet) | Helped |
| Remove EMA and use fixed class-specific thresholds (ablation) | ST-SGG with Motif · Visual Genome SGCls | mR@50 / 100: 16.9 / 18.0 -> 6.0 / 7.7 | Hurt |
| Remove class-specific momentum by setting coefficients to 0.5 (ablation) | ST-SGG with Motif · Visual Genome SGCls | mR@50 / 100: 16.9 / 18.0 -> 13.5 / 14.3 | Hurt |
| Remove EMA and use fixed class-specific thresholds (ablation) | ST-SGG with BGNN · Visual Genome SGCls | mR@50 / 100: 18.0 / 19.0 -> 13.9 / 14.8 | Hurt |
| Remove class-specific momentum by setting coefficients to 0.5 (ablation) | ST-SGG with BGNN · Visual Genome SGCls | mR@50 / 100: 18.0 / 19.0 -> 10.4 / 12.7 | Hurt |
| Add GSL to ST-SGG (ablation) | ST-SGG with BGNN · Visual Genome SGCls | mR@50 / 100: 18.0 / 19.0 -> 18.0 / 19.4 | Helped |
Surprising findings
- Fixed thresholding techniques from image recognition (constant, class-specific, frequency-weighted) degrade performance below the vanilla model in SGG self-training.
- Naive EMA-free fixed thresholding on Motif substantially lowers R@K and mR@K relative to vanilla Motif.
Limitations
- Mentions reliance on the threshold settings; sensitive to momentum hyperparameters alpha_inc and alpha_dec.
- Evaluated mainly on Visual Genome with results on OI-V6 only in appendix.
Experimental setup
Datasets: Visual Genome Open Images V6
Benchmarks: Visual Genome
Metrics: Recall@K mean Recall@K F@K
Baselines: Motif VCTree BGNN HetSGG DT2-ACBS PCPL KERN GBNet PE-Net Resampling TDE DLFE
Builds on: FixMatch Dash FlexMatch IE-Trans BGNN
Closest papers by meaning
- RA-SGG: Retrieval-Augmented Scene Graph Generation Framework via Multi-Prototype Learning (AAAI 2025 · similarity 0.90)
- Leveraging Predicate and Triplet Learning for Scene Graph Generation (CVPR 2024 · similarity 0.89)
- Semi-Supervised Clustering Framework for Fine-grained Scene Graph Generation (AAAI 2025 · similarity 0.89)
- LLM4SGG: Large Language Models for Weakly Supervised Scene Graph Generation (CVPR 2024 · similarity 0.88)
- Hydra-SGG: Hybrid Relation Assignment for One-stage Scene Graph Generation (ICLR 2025 · similarity 0.88)
- Scene Graph Generation Strategy with Co-occurrence Knowledge and Learnable Term Frequency (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.