Research map / Graph learning / Knowledge and temporal graphs
SCENIR: Visual Semantic Clarity through Unsupervised Scene Graph Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SCENIR, an unsupervised graph autoencoder with split GNN encoder, MLP decoders and adversarial regularization for image-to-image retrieval over scene graphs, evaluated against GED ground truth.
Problem
Image-to-image retrieval via scene graphs relies on supervised GNNs trained on caption-derived similarity labels, which are inconsistent across SBERT models; evaluation also depends on these unreliable caption-based rankings.
Approach
Replace caption-supervised GNNs with an unsupervised graph autoencoder (split VGAE-style encoder, MLP feature/edge decoders, adversarial latent regularization) that embeds scene graphs for cosine retrieval, and use approximate Graph Edit Distance as deterministic ground truth.
Key results
- SCENIR-GIN reaches NDCG@1 31.39 and MRR 59.01 on PSG vs. 29.83 and 58.73 for best supervised IRSGS-GIN variant (MRR 59.16 for IRSGS-GIN RoBERTa is higher), outperforming vision and VL models by a wide margin.
- On counterfactual retrieval (PSG), SCENIR achieves NDCG(B)@1 9.7 vs. 8.9 for IRSGS and 7.0 for GC.
- Total pipeline runtime reported as about 8 minutes versus about 50 minutes for IRSGS and about 3 hours for GC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SCENIR-GIN instead of IRSGS-GIN RoBERTa | PSG scene graph retrieval · PSG test | NDCG@3: 27.96 -> 28.77 | Helped |
| Use SCENIR-GIN instead of IRSGS-GIN RoBERTa | PSG scene graph retrieval · PSG test | MAP@3: 54.12 -> 54.16 | Helped |
| Use SCENIR-GIN instead of IRSGS-GIN RoBERTa | PSG scene graph retrieval · PSG test | MRR: 59.16 -> 59.01 | Hurt |
| Use SCENIR instead of IRSGS | Flickr30K retrieval with generated scene graphs · Flickr30K | NDCG@3: 21.02 -> 22.75 | Helped |
| Use SCENIR instead of IRSGS | Flickr30K retrieval with generated scene graphs · Flickr30K | MAP@3: 49.08 -> 50.31 | Helped |
| Use SCENIR instead of IRSGS | Flickr30K retrieval with generated scene graphs · Flickr30K | MRR: 54.30 -> 56.20 | Helped |
| Use SCENIR instead of IRSGS | counterfactual retrieval · PSG counterfactual retrieval | NDCG(B)@1: 8.9 -> 9.7 | Helped |
| Remove the MLP decoder (ablation) | SCENIR-GIN · PSG test | NDCG@3: 28.27 -> 24.95 | Hurt |
| Remove the discriminator (ablation) | SCENIR-GIN · PSG test | NDCG@3: 28.27 -> 24.37 | Hurt |
| Remove the GNN Edge Decoder (ablation) | SCENIR-GIN · PSG test | NDCG@3: 28.27 -> 25.81 | Hurt |
| Remove the GNN Feature Decoder (ablation) | SCENIR-GIN · PSG test | NDCG@3: 28.27 -> 25.68 | Hurt |
| Use split rather than unified encoder branches (ablation) | SCENIR encoder · PSG retrieval | retrieval accuracy | Helped |
Surprising findings
- Vision-only and VL models score roughly half as much NDCG as scene-graph GNNs on PSG retrieval.
- SBERT caption-based supervision choice (MPNet vs RoBERTa) produces disagreeing top-1 retrievals in at least 40% of cases.
Limitations
- Ground truth is approximate GED with hand-chosen cost functions, and the GED ground truth is itself a proxy for semantic similarity.
- In-the-wild evaluation depends on automatically generated scene graphs (PSGTR) and captions, which may be noisy.
- Experiments limited to PSG and Flickr30K with a single P100 GPU; ablations are on a subset of metrics.
Experimental setup
Datasets: PSG Flickr30K Visual Genome ImageNet-1K
Benchmarks: PSG Flickr30K PSG counterfactual retrieval
Metrics: NDCG@k MAP@k MRR
Baselines: ConvNeXt-V2 InceptionNeXt EfficientViT-L3 DeiT-III CLIP BLIP BLIP-2 ALBEF IRSGS VGAE ARVGA GC
Builds on: VGAE ARVGA Graph Feature Autoencoder IRSGS Graph Counterfactuals
Closest papers by meaning
- Scene Graph Embeddings Using Relative Similarity Supervision (AAAI 2021 · similarity 0.90)
- RA-SGG: Retrieval-Augmented Scene Graph Generation Framework via Multi-Prototype Learning (AAAI 2025 · similarity 0.87)
- Navigating the Unseen: Zero-shot Scene Graph Generation via Capsule-Based Equivariant Features (CVPR 2025 · similarity 0.87)
- Universal Scene Graph Generation (CVPR 2025 · similarity 0.87)
- Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning (CVPR 2025 · similarity 0.87)
- Instance-Level Composed Image Retrieval (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.