Research map / Graph learning / Knowledge and temporal graphs
HiKER-SGG: Hierarchical Knowledge Enhanced Robust Scene Graph Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HiKER-SGG, a scene graph generator that uses a automatically discovered hierarchical commonsense knowledge graph to predict superclass-then-subclass entity and predicate labels, and introduces the VG-C benchmark with 20 procedurally generated corruptions for robust SGG.
Problem
Scene graph generation assumes clean images; real-world corruptions (fog, snow, sun glare, water drops) degrade SGG performance, and existing SGG methods are not robust to them.
Approach
Build a hierarchical commonsense knowledge graph with superclass nodes (discovered by hierarchical clustering over GloVe semantic and MotifNet confusion-based pattern similarity) and use it to infer coarse superclass labels before fine subclass labels, with bridge edges to scene graph nodes and adaptive predicate refinement.
Key results
- On clean VG PredCls, HiKER-SGG reaches mR@100 UC/C of 69.2 / 41.2, versus EB-Net + EOA at 66.3 / 39.2.
- On VG-C PredCls, averaged over 20 corruptions, HiKER-SGG's mR@100 (C) is 35.1 with a 14.8% drop from clean, versus EB-Net's 31.0 with a 20.9% drop.
- Ablation: adding predicate hierarchical head gives +1.0% mR@k, entity hierarchy +0.5%, and adaptive refinement +0.2% to +0.8%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HiKER-SGG instead of EB-Net + EOA | SGG on Visual Genome · Visual Genome, PredCls | mR@100 UC/C: 66.3 / 39.2 -> 69.2 / 41.2 | Helped |
| Use HiKER-SGG instead of EB-Net + EOA | SGG on Visual Genome · Visual Genome, SGCls | mR@100 UC/C: 32.5 / 18.3 -> 36.7 / 21.4 | Helped |
| Use HiKER-SGG instead of EB-Net | SGG on corrupted images · VG-C, PredCls, average over 20 corruptions | mR@100 C: 56.7 (-14.5%) -> 61.4 (-11.3%) | Helped |
| Add adaptive refinement without hierarchical prediction (ablation) | EB-Net baseline configuration · Visual Genome, PredCls | mR@100 UC/C: 66.3 / 39.2 -> 67.1 / 39.8 | Helped |
| Add manually configured predicate hierarchy (ablation) | EB-Net baseline configuration · Visual Genome, PredCls | mR@100 UC/C: 66.3 / 39.2 -> 68.1 / 39.6 | Mixed |
| Add manually configured entity hierarchy to the predicate hierarchy (ablation) | EB-Net baseline configuration · Visual Genome, PredCls | mR@100 UC/C: 68.1 / 39.6 -> 68.2 / 39.7 | Helped |
| Add adaptive refinement to manually configured predicate and entity hierarchies (ablation) | EB-Net baseline configuration · Visual Genome, PredCls | mR@100 UC/C: 68.2 / 39.7 -> 68.7 / 40.0 | Helped |
| Replace manually configured hierarchies with hierarchies discovered by clustering (ablation) | HiKER-SGG · Visual Genome, PredCls | mR@100 UC/C: 68.2 / 39.7 -> 69.0 / 40.4 | Helped |
| Add adaptive refinement to discovered predicate and entity hierarchies (ablation) | HiKER-SGG · Visual Genome, PredCls | mR@100 UC/C: 69.0 / 40.4 -> 69.2 / 41.2 | Helped |
| Set adaptive-refinement α to 0.5 rather than 0.9 (ablation) | HiKER-SGG · Visual Genome, PredCls | mR@100 UC/C: 69.2 / 41.2 -> 66.9 / 40.0 | Hurt |
| Set adaptive-refinement α to 0.8 rather than 0.9 (ablation) | HiKER-SGG · Visual Genome, PredCls | mR@100 UC/C: 69.2 / 41.2 -> 68.5 / 40.7 | Hurt |
| Set adaptive-refinement α to 0.95 rather than 0.9 (ablation) | HiKER-SGG · Visual Genome, PredCls | mR@100 UC/C: 69.2 / 41.2 -> 69.1 / 40.9 | Hurt |
Surprising findings
- Hierarchical superclass-first prediction improves performance on clean images as well as corrupted ones, not only under corruption.
Limitations
- Corruptions are procedurally synthetic rather than collected from real adverse-condition images.
- Evaluated only on the Visual Genome SGG setting with a VGG-16 Faster R-CNN detector and older baselines.
- Results on clean SGCls are competitive but not uniformly best across all baselines in Table 1.
Experimental setup
Datasets: Visual Genome ConceptNet WordNet
Benchmarks: Visual Genome VG-C
Metrics: mR@20 mR@50 mR@100
Baselines: IMP+ Neural Motifs VCTree PCPL CogTree G2S GB-Net EB-Net + EOA PE-Net + SIL HML SQUAT
Builds on: GB-Net EB-Net MotifNet GGNN Hendrycks corruption benchmark
Closest papers by meaning
- Adaptive Visual Scene Understanding: Incremental Scene Graph Generation (NeurIPS 2024 · similarity 0.88)
- Towards Unbiased and Robust Spatio-Temporal Scene Graph Generation and Anticipation (CVPR 2025 · similarity 0.88)
- From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models (CVPR 2024 · similarity 0.87)
- Adaptive Self-training Framework for Fine-grained Scene Graph Generation (ICLR 2024 · similarity 0.87)
- Conformal Prediction and MLLM aided Uncertainty Quantification in Scene Graph Generation (CVPR 2025 · similarity 0.87)
- Hydra-SGG: Hybrid Relation Assignment for One-stage Scene Graph Generation (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.