Research map / Label-efficient and robust learning / Recognition and long-tail

RA-SGG: Retrieval-Augmented Scene Graph Generation Framework via Multi-Prototype Learning

AAAI 2025 · Label-efficient and robust learning › Recognition and long-tail · approach: Contrastive learning for imbalanced data · 7 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Reframes scene graph generation as multi-label classification with partial annotation and augments single predicate labels with retrieved, inverse-propensity-sampled pseudo-labels trained via multi-prototype learning on PE-Net.

Problem

Scene graph generation suffers from long-tailed predicate distributions and semantic ambiguity between predicates, causing bias toward head predicates; each relation is annotated with only a single predicate even when multiple predicates apply.

Approach

Retrieve semantically similar relation instances from a frozen memory bank built from a pre-trained SGG model, flag likely multi-labeled instances via label inconsistency with retrieved predicates, augment labels by sampling retrieved predicates weighted by inverse propensity, and train with multi-prototype loss so embeddings reflect several predicates.

Key results

• On VG PredCls, RA-SGG with PE-Net reaches F@50/100 of 45.7/48.6 versus 42.4/45.0 for PE-Net, with mR@50/100 rising from 31.5/33.8 to 36.2/39.1.
• On GQA PredCls, PE-Net+RA-SGG improves F@50/100 to 40.9/42.4 from 35.4/36.5 for PE-Net.
• Human evaluation of retrieval accuracy on 100 sampled instances gives 84.20%; training time is about 1.3x PE-Net with unchanged inference time.

Measured effects reported in the paper

Surprising findings

• Feature-level augmentation (Feat Aug) and mix-up raise R@K but reduce mR@K, which the authors attribute to shortcuts toward head predicates.
• Retrieval without reliable multi-label selection ('w/o select.') still improves over PE-Net on mR@K and F@K.

Limitations

• The memory bank is restricted to training instances; external data sources are not used.
• Only applied on the PE-Net backbone in the paper.
• Retrieval accuracy is assessed by human evaluation on only 100 sampled instances.

Experimental setup

Datasets: Visual Genome GQA

Benchmarks: Visual Genome GQA

Metrics: R@K mR@K F@K

Baselines: PE-Net IETrans CFA ST-SGG NICE GCL SQUAT HetSGG BGNN KERN

Builds on: PE-Net IE-Trans Motif

Closest papers by meaning

• Adaptive Self-training Framework for Fine-grained Scene Graph Generation (ICLR 2024 · similarity 0.90)
• Leveraging Predicate and Triplet Learning for Scene Graph Generation (CVPR 2024 · similarity 0.89)
• LLM4SGG: Large Language Models for Weakly Supervised Scene Graph Generation (CVPR 2024 · similarity 0.89)
• Semi-Supervised Clustering Framework for Fine-grained Scene Graph Generation (AAAI 2025 · similarity 0.89)
• From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models (CVPR 2024 · similarity 0.88)
• Noise-Guided Predicate Representation Extraction and Diffusion-Enhanced Discretization for Scene Graph Generation (ICML 2025 · similarity 0.88)