Research map / Graph learning / Knowledge and temporal graphs
Adaptive Visual Scene Understanding: Incremental Scene Graph Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces a continual scene graph generation (CSEGG) benchmark with three learning regimes on Visual Genome and RAS, a generative replay method that synthesizes replay images from recomposed triplet labels using Stable Diffusion instead of storing images.
Problem
Scene graph generation models must continually learn new objects and relationships without forgetting old ones, a setting that existing continual learning work for classification and detection does not address, and long-tailed distributions and combinatorial relation complexity make it harder.
Approach
Replay via analysis by synthesis: store only triplet labels and the previous model snapshot, recompose triplets into context-coherent text prompts (BERT embeddings plus agglomerative clustering), synthesize replay images with Stable Diffusion, pseudo-label them with the frozen old SGG model, and apply triplet-level long-tailed sampling.
Key results
- RAS outperforms all CSEGG baselines in scenarios S1 and S2, with Replay@20% storing ~2 GB versus RAS ~1.2 GB of storage.
- In S2, all CSEGG models including Replay@100% fall well below the Joint upper bound; RAS leads among baselines but remains far from Joint.
- Naive baseline performs worst in all settings, showing significant catastrophic forgetting; directly combining EWC or PackNet with SGG backbones underperforms replay.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use RAS instead of Replay@10% and Naive | SGTR-based CSEGG methods · S3 unknown-object generalization test set | Gen R | Helped |
| use RAS instead of Replay@10% and Naive | SGTR-based CSEGG methods · S3 unknown-object generalization test set | Gen R_bbox | Hurt |
| use Replay@10% instead of Naive | SGTR-based CSEGG methods · S3 unknown-object generalization test set | Gen R | Helped |
| use Replay@10% instead of Naive | SGTR-based CSEGG methods · S3 unknown-object generalization test set | Gen R_bbox | Hurt |
| remove the context checker and randomly combine triplet labels in prompts (ablation) | RAS · S1 and S2 | evaluation metrics: approximately 2% decrease | Hurt |
| remove LTD sampling from replay triplets (ablation) | RAS · S1 and S2 | evaluation metrics: absolute decrease of 1-2% | Hurt |
| replace multiple triplet labels per prompt with a single triplet label (ablation) | RAS · S1 and S2 | evaluation metrics: approximately 3% decrease | Hurt |
| increase the number of generated images per prompt (γ varied from 2 to 8) (ablation) | RAS · S1 and S2 | evaluation metrics | Helped |
| fine-tune DETR over S1 tasks | SGTR-based CSEGG methods · Visual Genome test set | mAP: a slight decrease of 0.4 | No clear effect |
| apply BLS@Replay@10% instead of LVIS@Replay@10% | Replay-based CSEGG methods · S1 | performance | Helped |
Surprising findings
- Removing the context checker from RAS lowers performance by about 2% across metrics, but the paper notes out-of-context prompts may be hard for Stable Diffusion.
- BLS@Replay@10% significantly outperforms LVIS@Replay@10%, contrary to classical SGG findings where BLS is considered more effective than LVIS.
Limitations
- Pseudo-labels for synthesized replay images come from the preceding SGG model, so errors can propagate across tasks.
- A large performance gap to the joint training upper bound persists in scenario S2.
- Experiments use only static images under i.i.d. assumptions, not video streams.
Experimental setup
Datasets: Visual Genome
Benchmarks: CSEGG benchmark (Visual Genome-based)
Metrics: R@K mR@K Forgetfulness (F@K) Avg.R@K FWT@K BWT@K Gen R_bbox@K Gen R@K
Baselines: Naive EWC PackNet Replay@10% Replay@20% Replay@100% Joint Training RAS_GT LVIS BLS
Builds on: SGTR Replay EWC PackNet Stable Diffusion
Closest papers by meaning
- Scene Graph Generation Strategy with Co-occurrence Knowledge and Learnable Term Frequency (ICML 2024 · similarity 0.89)
- From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models (CVPR 2024 · similarity 0.89)
- HiKER-SGG: Hierarchical Knowledge Enhanced Robust Scene Graph Generation (CVPR 2024 · similarity 0.88)
- Scene Graph-Grounded Image Generation (AAAI 2025 · similarity 0.88)
- EGTR: Extracting Graph from Transformer for Scene Graph Generation (CVPR 2024 · similarity 0.88)
- DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.