Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Navigating the Unseen: Zero-shot Scene Graph Generation via Capsule-Based Equivariant Features
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CAPSGG, a capsule-network-based scene graph generation model with a three-stream (visual, semantic, spatial) pipeline and GroupLoss to improve zero-shot predicate classification on Visual Genome.
Problem
Zero-shot scene graph generation: predicting unseen subject-predicate-object triplets, where traditional models fail to capture the equivariant transformations of predicate features needed to generalize to unseen combinations.
Approach
Model predicates as capsules in a high-dimensional space so that unseen predicates are transformations of seen ones; combine visual, text and spatial modality capsules via self-attention routing, and replace the global MarginLoss with a group-wise loss over same-label capsules to balance invariant and equivariant features and mitigate long-tail effects.
Key results
- CAPSGG reaches 29.3 zR@20 on PredCls without graph constraint versus 27.4 for T-CAR, and 68.0 zR@100 without constraint.
- On SGCls with graph constraint, zero-shot recall improves up to 132.26% over T-CAR (zR@50 28.8 vs 12.4 without constraint).
- Removing GroupLoss drops PredCls zR@100 from 45.2 to 35.9 in the ablation table (with GroupLoss 45.2 vs 35.9 without).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CAPSGG instead of T-CAR | SGG · VG150 PredCls, no graph constraint | zR@100: 50.8 -> 68.0 | Helped |
| use CAPSGG instead of T-CAR | SGG · VG150 SGCls, no graph constraint | zR@50: 12.4 -> 28.8 | Helped |
| use CAPSGG instead of T-CAR | SGG · VG150 SGDet, no graph constraint | zR@50: 6.2 -> 12.5 | Helped |
| use CAPSGG instead of T-CAR | SGG · VG150 PredCls, graph constraint | zR@100: 34.9 -> 46.1 | Helped |
| use CAPSGG instead of T-CAR | SGG · VG150 SGCls, graph constraint | zR@100: 10.6 -> 24.3 | Helped |
| use CAPSGG instead of T-CAR | SGG · VG150 SGDet, graph constraint | zR@100: 6.0 -> 12.8 | Helped |
| replace three-branch CFE capsule construction with direct feature concatenation (ablation) | CAPSGG · VG150 Predicate classification | zR@50: 37.7 -> 33.3 | Hurt |
| remove any modality branch from the Three-Stream Pipeline (ablation) | CAPSGG · VG150 zero-shot SGG | zR@K | Hurt |
| remove GroupLoss and use the original MarginLoss (ablation) | CAPSGG · VG150 PredCls and SGCls | zR@K | Hurt |
| set GroupLoss λ to 0.5 instead of 0.15 (ablation) | CAPSGG · VG150 Predicate classification | zR@50: 20.0 -> 20 | No clear effect |
| set GroupLoss λ to 0.1 instead of 0.15 (ablation) | CAPSGG · VG150 Predicate classification | zR@50: 20.0 -> 18.2 | Hurt |
| set GroupLoss λ to 0.05 instead of 0.15 (ablation) | CAPSGG · VG150 Predicate classification | zR@50: 20.0 -> 17.2 | Hurt |
Surprising findings
- Unlike T-CAR, adding the text (semantic label) branch improves zero-shot performance in this model.
- Excessive weight on the negative term in the margin-style loss causes 'metric inversion', where no-graph-constraint metrics fall below graph-constrained ones.
Limitations
- Code is promised only upon publication; no release at submission time.
- Hyperparameter lambda is sensitive, and poor settings cause metric inversion.
- Zero-shot gains are reported only on VG150 with a single unified protocol.
Experimental setup
Datasets: Visual Genome VG150
Benchmarks: Visual Genome VG150
Metrics: zR@K ng-zR@K R@K
Baselines: IMP IMP++ Motifs VTransE TDE BGNN UVTransE EBM KGC RGN COACHER T-CAR
Builds on: CapsNet (dynamic routing) T-CAR MarginLoss Faster R-CNN
Closest papers by meaning
- Leveraging Predicate and Triplet Learning for Scene Graph Generation (CVPR 2024 · similarity 0.88)
- LLM4SGG: Large Language Models for Weakly Supervised Scene Graph Generation (CVPR 2024 · similarity 0.88)
- Universal Scene Graph Generation (CVPR 2025 · similarity 0.88)
- SCENIR: Visual Semantic Clarity through Unsupervised Scene Graph Retrieval (ICML 2025 · similarity 0.87)
- Multi-Prototype Space Learning for Commonsense-Based Scene Graph Generation (AAAI 2024 · similarity 0.87)
- TPR: Topology-Preserving Reservoirs for Generalized Zero-Shot Learning (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.